andrebrait

I had the crash yesterday without the proxy, so now I tried to get everything off the docker socket. Let's see... Let me check your logs.

It's still running rock solid. I also moved it back to a docker folder as that's more convenient and faster for me and so far I have not experienced a single crash. So yes, the result seems to be that linuxserver's socket-proxy was causing dockerd to crash.

No crashes since removing the linuxserver.io's socket-proxy containers. Gonna try tecnativa or wollomatic and see what that gets me.

Yes. I removed it and I'm now just mounting the docker socket onto Traefik and Authentik. Since the crash pretty much always happens in a matter of days, if this setup endures (or if I switch to a different proxy, like the original Tecnativa or the one by wollomatic), then I'll report it as a bug to the linuxserver socket-proxy devs.

@JorgeB here's the diagnostics after it just happened. So far it's been reproducible in btrfs + folder, btrfs + image and xfs + image. I think it might be that the socket proxy container might be causing the docker daemon to die somehow. nas-diagnostics-20260311-0206.zip

The one I posted in the first post was after it had happened. I'll get a second one, but I switched to a XFS image. Let's see how that goes...

Using a btrfs image, same result. After a while, it goes bad. Docker becomes unavailable. Running dockerd manually then nets me this (attached) dockerd.txt

Folder in a btrfs mirror pool. I've now switched to an image. Will update this thread if I have similar issues. And disabling the pivot root is actually very insecure, so I scratched that idea

@JorgeB after running memtest86+ with 8 passes and attesting that the RAM settings as dictated by Intel and sadly not followed by ASUS is indeed stable, I ran into the issue again and found an interesting log line when trying to run dockerd manually: ERRO[2026-03-09T23:47:27.577083533+01:00] failed to start container container=14461b57ce409f4063816b7724d63faedfacd3a0476266e0cfa838847dc85a22 error="failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error duringWhich led me to this: GitHubDocker report "error jailing process inside rootfs: pivot...Description I packged a ubuntu22.04.5_aarch64 system to ramdisk file ,boot from PXE, installed docker 28.3.3, and then run "docker run hello-world", a issue happpend as below: root@node1:~# docker ...And sure as heck, setting DOCKER_RAMDISK=1 and starting dockerd seems to make it work. Now I need a way to always pass that onto docker when started by unRAID. I'm not sure this would be an actual solution long-term, but maybe docker has a nice surprise in the recent releases (and I'm not sure whether the unRAID devs know about this): https://docs.docker.com/engine/storage/containerd

@JorgeB my RAM failed after the 3rd and 5th passes of memtest86+ (ver 8.00, so a bit newer than what ships with unRAID, though I don't think this matters, but mentioning it for the sake of completeness). It seems the guideline out there is that memtest86+ requires at least 8 passes to be truly indicative of stability. Since my motherboard runs my RAM and the memory controller at non-factory specs by default (go figure...) I'm now trying that to see if the source of the instability is that or the RAM itself. CPU: Intel Core i7-11700 Motherboard: ASUS PRIME Z590M-PLUS RAM: 2x 32GB Crucial (Micron) DDR4-3200 CT32G4DFD832A.M16FF For the 11th gen, Intel dictates only their i9 K/KF CPUs can run DDR4-3200 at 1:1 RAM-to-memory-controller ratio. The other CPUs of the 11th gen either run 2:1 ratio for DDR4-3200 and above, or 1:1 ratio for DDR4-2933 and below. However, most motherboards seem to default to 1:1 even at DDR4-3200 speeds which might be "stable" for desktop usage, but perhaps not stable enough for a 24/7 unRAID server. I've manually set the speed to DDR4-2933 and the ratio to 1:1 in the BIOS and I'll see where that leads us.

Btw, I don't think this is a folder issue. I guess it's RAM yet again. It would be the second batch of Crucial RAM I get (with Micron ICs, of course) that manage to pass every conceivable memory test, but crash the system at random.

I get that it might indeed be the case, but think about it... You are storing an image file... In a file system... With a file system inside... And you're mounting that file system... So every write, mount, etc., go through the file system inside the image and then the file system in the actual storage medium... If anything, given I'm using btrfs (which is the default filesystem for images in Docker in unRAID), it should only increase the chances of some weird stuff happening, because all the image would accomplish is writing to btrfs twice (once in the image, which then gets written to my btrfs filesystem) instead of once (the filesystem itself). So either it's some btrfs bug that's being accidentally worked around by using the image (which is a "clean" btrfs filesystem); or there's something really funky going on specifically with unRAID or how Docker is being built there. I run Docker on a lot of distributions and they're all just using standard folders. That includes production deployments at enterprise scale 😅 Well, admittedly, none of them run Slackware, so who knows, but it's quite weird from any point of view to think the folder can cause issues like this. Maybe some order of initialization stuff is somehow borked? Idk, but I guess the unRAID devs really should take a look at this.

The only thing I haven't tried is switching to an i Even when using bog standard stuff like a regular btrfs mirror/single disk? I wonder why that would be the case, if so, as a docker folder is the default way of doing it for any Linux-based system I've ever seen other than unRAID. The image is just mounted as a volume containing the filesystem, after all, isn't it? I'd understand if there was a difference in the driver or something else, but it sounds more like there's some weird bug in either a filesystem or something else that is "accidentally" worked around when using an image, I guess?

Sometime overnight, but this is also not the first time I ever see it happen. I've tried to debug things myself quite a number of days before coming to this. I got a new memory module I'm going to test, but I'm not confident it'll solve anything. This particular machine is composed of the motherboard my previous unRAID build used + the CPU and memory my proxmox host used, but I've started using a single RAM stick to try and rule out the RAM being to blame here.

I've been facing a lot of weird issues lately. What I tried and didn't result in much of an improvement: Changing the docker folder from a ZFS volume to a btrfs volume Changing RAM (currently running a single stick JEDEC DDR4-3200 from Crucial, passed memtest86+ 3 times, Live Memory Tester plugin 2 loops, etc.) I originally bumped into this while trying to upgrade some docker containers and getting error messages about not being able to delete images. Suspecting being a bug in the overlay2 + ZFS combination, I switched to a btrfs mirror, but I got the same result after a while. Symptoms (in the order I notice them) Suddenly some containers crash. Others keep running (luckily Traefik being one of them, whereas the Crowdsec or Redis container living in the same 2. Compose stack die, which led me to being able to see the Crowdsec plugin's error page when trying to access anything I host behind Traefik) docker ps fails due to being unable to access the docker socket. "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?" Accessing the Docker tab in the webUI gets me a message "Docker failed to start" and nothing is visible Going to Settings -> Docker and disabling Docker results in it saving the changes forever. If I refresh the page, it says Docker is still running despite being disabled. Trying to enable Docker again works, but all containers have their auto start setting disabled now (?) Trying to start some of the not-started containers (not Compose, just regular unRAID templates via the GUI) gets me "Image can not be deleted, in use by other container(s)" Trying to stop specific containers and/or via docker compose results in errors. Not all containers, just some. See log line below when trying to "docker compose down" my nextcloud stack: panic: jsonschema: Compile("compose-spec.json"): json-pointer in "file:///boot/config/plugins/compose.manager/projects/nextcloud/compose-spec.json#/definitions/deployment/patternProperties/%5Ex-" not found goroutine 1 [running]: github.com/santhosh-tekuri/jsonschema/v6.(*Compiler).MustCompile(0x253c680?, {0x29307c3, 0x11}) github.com/santhosh-tekuri/jsonschema/[email protected]/compiler.go:172 +0xb9 github.com/compose-spec/compose-go/v2/schema.Validate(0xc000391170) github.com/compose-spec/compose-go/[email protected]/schema/schema.go:63 +0x325 github.com/compose-spec/compose-go/v2/loader.loadYamlFile.func1({0x2532080?, 0xc0006c0330?}, {0xc0006e69a8, 0x1, 0x1}) github.com/compose-spec/compose-go/[email protected]/loader/loader.go:481 +0x3ee github.com/compose-spec/compose-go/v2/loader.loadYamlFile({0x2d37900, 0xc0003a9720}, {{0xc00005c0c0, 0x53}, {0xc00050d200, 0x474, 0x475}, 0x0}, 0xc00068a360, {0xc000632280, ...}, ...) github.com/compose-spec/compose-go/[email protected]/loader/loader.go:517 +0x53d github.com/compose-spec/compose-go/v2/loader.loadYamlModel({0x2d37900, 0xc0003a9720}, {{0x0, 0x0}, {0xc000632280, 0x37}, {0xc00021c6c0, 0x2, 0x2}, 0xc000390060}, ...) github.com/compose-spec/compose-go/[email protected]/loader/loader.go:378 +0x19e github.com/compose-spec/compose-go/v2/loader.load({0x2d37900, 0xc0003a9720}, {{0x0, 0x0}, {0xc000632280, 0x37}, {0xc00021c6c0, 0x2, 0x2}, 0xc000390060}, ...) github.com/compose-spec/compose-go/[email protected]/loader/loader.go:538 +0x2f9 github.com/compose-spec/compose-go/v2/loader.loadModelWithContext({0x2d37900, 0xc0003a9720}, 0xc000694800, 0xc00068a360) github.com/compose-spec/compose-go/[email protected]/loader/loader.go:350 +0x105 github.com/compose-spec/compose-go/v2/loader.LoadWithContext({0x2d37900, 0xc0003a9720}, {{0x0, 0x0}, {0xc000632280, 0x37}, {0xc00021c6c0, 0x2, 0x2}, 0xc000390060}, ...) github.com/compose-spec/compose-go/[email protected]/loader/loader.go:326 +0xd8 github.com/compose-spec/compose-go/v2/cli.(*ProjectOptions).LoadProject(0xc00057a000, {0x2d37900, 0xc0003a9720}) github.com/compose-spec/compose-go/[email protected]/cli/options.go:455 +0x119 github.com/docker/compose/v2/cmd/compose.(*ProjectOptions).ToProject(0xc00040bcc0, {0x2d37900, 0xc0003a9720}, {0x2d567b8, 0xc0001ad180}, {0x438ff00, 0x0, 0x0}, {0xc0001116b0, 0x2, ...}) github.com/docker/compose/v2/cmd/compose/compose.go:326 +0x53f github.com/docker/compose/v2/cmd/compose.(*ProjectOptions).projectOrName(0xc000116df0?, {0x2d37900?, 0xc0003a9720?}, {0x2d567b8?, 0xc0001ad180?}, {0x438ff00?, 0x6c6f72?, 0xc000111748?}) github.com/docker/compose/v2/cmd/compose/compose.go:238 +0x8a github.com/docker/compose/v2/cmd/compose.runStop({0x2d37900, 0xc0003a9720}, {0x2d567b8?, 0xc0001ad180?}, {0x2d694b8, 0xc00062dcc0}, {0xc00040bcc0?, 0x18?, 0x47c032?}, {0x438ff00, ...}) github.com/docker/compose/v2/cmd/compose/stop.go:57 +0x88 github.com/docker/compose/v2/cmd/compose.stopCommand.func2({0x2d37900?, 0xc0003a9720?}, {0x438ff00?, 0xc000002380?, 0x2147a8a?}) github.com/docker/compose/v2/cmd/compose/stop.go:46 +0x65 github.com/docker/compose/v2/cmd/compose.stopCommand.Adapt.func3({0x2d37900?, 0xc0003a9720?}, 0x2?, {0x438ff00?, 0x2d16c50?, 0x9d0d69?}) github.com/docker/compose/v2/cmd/compose/compose.go:128 +0x30 github.com/docker/compose/v2/cmd/compose.stopCommand.Adapt.AdaptCmd.func5(0xc000415508, {0x438ff00, 0x0, 0x0}) github.com/docker/compose/v2/cmd/compose/compose.go:112 +0x143 github.com/docker/cli/cli-plugins/plugin.RunPlugin.func1.1.2(0xc000415508, {0x438ff00, 0x0, 0x0}) github.com/docker/[email protected]+incompatible/cli-plugins/plugin/plugin.go:65 +0x64 github.com/docker/compose/v2/cmd/cmdtrace.Setup.wrapRunE.func2(0xc000415508?, {0x438ff00?, 0x0?, 0x0?}) github.com/docker/compose/v2/cmd/cmdtrace/cmd_span.go:87 +0x63 github.com/spf13/cobra.(*Command).execute(0xc000415508, {0xc00062a0d0, 0x0, 0x0}) github.com/spf13/[email protected]/command.go:1015 +0xaaa github.com/spf13/cobra.(*Command).ExecuteC(0xc00047d208) github.com/spf13/[email protected]/command.go:1148 +0x46f github.com/spf13/cobra.(*Command).Execute(...) github.com/spf13/[email protected]/command.go:1071 github.com/docker/cli/cli-plugins/plugin.RunPlugin(0xc0001ad180, 0xc000414608, {{0x2919a7b, 0x5}, {0x2923d27, 0xb}, {0x2cff8a0, 0x7}, {0x0, 0x0}, ...}) github.com/docker/[email protected]+incompatible/cli-plugins/plugin/plugin.go:80 +0x145 github.com/docker/cli/cli-plugins/plugin.Run(0x2a843f8, {{0x2919a7b, 0x5}, {0x2923d27, 0xb}, {0x2cff8a0, 0x7}, {0x0, 0x0}, {0x0, ...}}, ...) github.com/docker/[email protected]+incompatible/cli-plugins/plugin/plugin.go:99 +0x145 main.pluginMain() github.com/docker/compose/v2/cmd/main.go:38 +0x14f main.main() github.com/docker/compose/v2/cmd/main.go:81 +0x19eI'm attaching a diagnostics file from when the crash happens and after trying to restore Docker to a working state. nas-diagnostics-20260304-0629.zip nas-diagnostics-20260304-0805.zip

Yes, you should be able to delete them. If yoh want, you can "test that" by renaming the folder / moving the contents somewhere else, rebooting/restarting docker and seeing if anything changes for you. But given you're using the image method, you should be able to just delete that folder as it's not being used anymore anyway.

Mine is crashing with a segmentation fault, repeatedly, as soon as I try to do anything. I've reinstalled, but to no avail. I'm now just using my firewall to route traffic to between my nodes, since Tailscale on it and on my backup node (also unRAID, same version and all) is running fine. tailscale.log.txt

Just had the same. It would also prematurely end the transfer while there were still 90GB left.

@Fuzzy0101 after further investigating, I noticed that: It always happens when I apply settings It stops happening when I pass pcie_aspm=force to the kernel It stops happening when I enable Native ASPM in the UEFI So I think what's happening here is that it cannot write to /sys/module/pcie_aspm/parameters/policy. Under some circumstances, this is not writable, such as when the BIOS is controlling ASPM. I think you should add 1) log a better error message when this fails to be written, and 2) grey out the field for the plugin's page and an explanation of why it cannot be triggered. Lastly, I'd like to ask you to add a caution message for the Powersave option for SATA. This is known to cause data corruption with some hardware combinations out there, so it would be best if there was a clear warning. Also, I think the Power Profile option is changing /sys/devices/system/cpu/cpu*/cpufreq/energy_performance_preference, right? Does it also do anything for /sys/devices/system/cpu/cpu*/power/energy_perf_bias ? It's essentially the same, but with less values and it might be used when the first is not available, and uses/accepts the following values (both strings and numbers are accepted): “performance” = 0, “balance-performance” = 4, “normal”= 6, “balance-power”= 8, “power” = 15. ps.: it uses - instead of _ , unliked /sys/devices/system/cpu/cpu*/cpufreq/energy_performance_preference, which takes one of "performance", "balance_performance", "balance_power power".

As for the SATA power management options, it would be probably better to have a clear warning about min_power in particular, since it's a riskier option, according to the Arch Wiki

Only during boot, as far as I can tell. As for network card, the only NIC enabled is an Intel X710-DA2. Here are the logs from applying settings: Jul 22 02:30:13 NAS ool www[1128898]: /usr/local/emhttp/plugins/autotweak/scripts/autotweak 'apply' Jul 22 02:30:13 NAS autotweak: Appply powersupersave profile Jul 22 02:30:13 NAS autotweak: Set scaling governor to powersave for CPU 0 Jul 22 02:30:13 NAS autotweak: Set scaling governor to powersave for CPU 1 Jul 22 02:30:13 NAS autotweak: Set scaling governor to powersave for CPU 10 Jul 22 02:30:13 NAS autotweak: Set scaling governor to powersave for CPU 11 Jul 22 02:30:13 NAS autotweak: Set scaling governor to powersave for CPU 12 Jul 22 02:30:13 NAS autotweak: Set scaling governor to powersave for CPU 13 Jul 22 02:30:13 NAS autotweak: Set scaling governor to powersave for CPU 14 Jul 22 02:30:13 NAS autotweak: Set scaling governor to powersave for CPU 15 Jul 22 02:30:13 NAS autotweak: Set scaling governor to powersave for CPU 2 Jul 22 02:30:13 NAS autotweak: Set scaling governor to powersave for CPU 3 Jul 22 02:30:13 NAS autotweak: Set scaling governor to powersave for CPU 4 Jul 22 02:30:13 NAS autotweak: Set scaling governor to powersave for CPU 5 Jul 22 02:30:13 NAS autotweak: Set scaling governor to powersave for CPU 6 Jul 22 02:30:13 NAS autotweak: Set scaling governor to powersave for CPU 7 Jul 22 02:30:13 NAS autotweak: Set scaling governor to powersave for CPU 8 Jul 22 02:30:13 NAS autotweak: Set scaling governor to powersave for CPU 9 Jul 22 02:30:13 NAS autotweak: Set performance preference to power for CPU 0 Jul 22 02:30:13 NAS autotweak: Set performance preference to power for CPU 1 Jul 22 02:30:13 NAS autotweak: Set performance preference to power for CPU 10 Jul 22 02:30:13 NAS autotweak: Set performance preference to power for CPU 11 Jul 22 02:30:13 NAS autotweak: Set performance preference to power for CPU 12 Jul 22 02:30:13 NAS autotweak: Set performance preference to power for CPU 13 Jul 22 02:30:13 NAS autotweak: Set performance preference to power for CPU 14 Jul 22 02:30:13 NAS autotweak: Set performance preference to power for CPU 15 Jul 22 02:30:13 NAS autotweak: Set performance preference to power for CPU 2 Jul 22 02:30:13 NAS autotweak: Set performance preference to power for CPU 3 Jul 22 02:30:13 NAS autotweak: Set performance preference to power for CPU 4 Jul 22 02:30:13 NAS autotweak: Set performance preference to power for CPU 5 Jul 22 02:30:13 NAS autotweak: Set performance preference to power for CPU 6 Jul 22 02:30:13 NAS autotweak: Set performance preference to power for CPU 7 Jul 22 02:30:13 NAS autotweak: Set performance preference to power for CPU 8 Jul 22 02:30:13 NAS autotweak: Set performance preference to power for CPU 9 Jul 22 02:30:13 NAS autotweak: Set HWP Dynamic Boost to 1 Jul 22 02:30:13 NAS autotweak: Set no_turbo to 0 Jul 22 02:30:13 NAS autotweak: Failed to set ASPM policy Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:00:00.0 Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:00:01.0 Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:00:02.0 Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:00:08.0 Jul 22 02:30:13 NAS autotweak: Skipping device 0000:00:14.0 with banned driver: xhci_hcd Jul 22 02:30:13 NAS autotweak: Skipping device 0000:00:16.0 with banned driver: mei_me Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:00:17.0 Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:00:1b.0 Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:00:1b.2 Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:00:1b.3 Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:00:1b.4 Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:00:1c.0 Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:00:1c.4 Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:00:1d.0 Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:00:1f.0 Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:00:1f.2 Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:00:1f.4 Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:01:00.0 Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:05:00.0 Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:07:00.0 Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:07:00.1 Jul 22 02:30:13 NAS autotweak: PCIe PM control set to 'auto' for device 0000:08:00.0 Jul 22 02:30:13 NAS autotweak: PM control set to 'auto' for ata device ata1 Jul 22 02:30:13 NAS autotweak: PM control set to 'auto' for ata device ata2 Jul 22 02:30:13 NAS autotweak: PM control set to 'auto' for ata device ata3 Jul 22 02:30:13 NAS autotweak: PM control set to 'auto' for ata device ata4 Jul 22 02:30:13 NAS autotweak: PM control set to 'auto' for ata device ata5 Jul 22 02:30:13 NAS autotweak: PM control set to 'auto' for ata device ata6 Jul 22 02:30:13 NAS autotweak: Set SATA link power management to med_power_with_dipm for host2 Jul 22 02:30:13 NAS autotweak: Set SATA link power management to med_power_with_dipm for host3 Jul 22 02:30:13 NAS kernel: ahci 0000:00:17.0: port does not support device sleep Jul 22 02:30:13 NAS autotweak: Set SATA link power management to med_power_with_dipm for host4 Jul 22 02:30:13 NAS kernel: ahci 0000:00:17.0: port does not support device sleep Jul 22 02:30:13 NAS autotweak: Set SATA link power management to med_power_with_dipm for host5 Jul 22 02:30:13 NAS kernel: ahci 0000:00:17.0: port does not support device sleep Jul 22 02:30:13 NAS autotweak: Set SATA link power management to med_power_with_dipm for host6 Jul 22 02:30:13 NAS kernel: ahci 0000:00:17.0: port does not support device sleep Jul 22 02:30:13 NAS autotweak: Set SATA link power management to med_power_with_dipm for host7 Jul 22 02:30:13 NAS kernel: ahci 0000:00:17.0: port does not support device sleep Jul 22 02:30:13 NAS autotweak: Set nvme0 APST table entry 0 Next state: 3 Timeout: 50 Jul 22 02:30:13 NAS autotweak: Set nvme0 APST table entry 1 Next state: 3 Timeout: 50 Jul 22 02:30:13 NAS autotweak: Set nvme0 APST table entry 2 Next state: 3 Timeout: 50 Jul 22 02:30:13 NAS autotweak: Set nvme0 APST table entry 3 Next state: 4 Timeout: 500 Jul 22 02:30:14 NAS autotweak: Set nvme0 APST policy to powersupersave Jul 22 02:30:14 NAS autotweak: Set nvme1 APST table entry 0 Next state: 3 Timeout: 50 Jul 22 02:30:14 NAS autotweak: Set nvme1 APST table entry 1 Next state: 3 Timeout: 50 Jul 22 02:30:14 NAS autotweak: Set nvme1 APST table entry 2 Next state: 3 Timeout: 50 Jul 22 02:30:14 NAS autotweak: Set nvme1 APST table entry 3 Next state: 4 Timeout: 500 Jul 22 02:30:14 NAS autotweak: Set nvme1 APST policy to powersupersave Jul 22 02:30:14 NAS autotweak: Banned CPUs for network interface interrupt affinity: 0,12 Jul 22 02:30:14 NAS autotweak: Usable CPUs for eth0: 1-11,13-15 Jul 22 02:30:14 NAS autotweak: Set CPU affinity of interrupt 191 and tx queue 0 of eth0 to CPU 1 Jul 22 02:30:14 NAS autotweak: Set CPU affinity of interrupt 192 and tx queue 1 of eth0 to CPU 2 Jul 22 02:30:14 NAS autotweak: Set CPU affinity of interrupt 193 and tx queue 2 of eth0 to CPU 3 Jul 22 02:30:14 NAS autotweak: Set CPU affinity of interrupt 194 and tx queue 3 of eth0 to CPU 4 Jul 22 02:30:14 NAS autotweak: Set CPU affinity of interrupt 195 and tx queue 4 of eth0 to CPU 5 Jul 22 02:30:14 NAS autotweak: Set CPU affinity of interrupt 196 and tx queue 5 of eth0 to CPU 6 Jul 22 02:30:14 NAS autotweak: Set CPU affinity of interrupt 197 and tx queue 6 of eth0 to CPU 7 Jul 22 02:30:14 NAS autotweak: Set CPU affinity of interrupt 198 and tx queue 7 of eth0 to CPU 8 Jul 22 02:30:14 NAS autotweak: Set CPU affinity of interrupt 199 and tx queue 8 of eth0 to CPU 9 Jul 22 02:30:14 NAS autotweak: Set CPU affinity of interrupt 200 and tx queue 9 of eth0 to CPU 10 Jul 22 02:30:14 NAS autotweak: Set CPU affinity of interrupt 201 and tx queue 10 of eth0 to CPU 11 Jul 22 02:30:14 NAS autotweak: Set CPU affinity of interrupt 202 and tx queue 11 of eth0 to CPU 13 Jul 22 02:30:14 NAS autotweak: Set CPU affinity of interrupt 203 and tx queue 12 of eth0 to CPU 14 Jul 22 02:30:15 NAS autotweak: Set CPU affinity of interrupt 204 and tx queue 13 of eth0 to CPU 15 Jul 22 02:30:15 NAS autotweak: Set CPU affinity of interrupt 205 and tx queue 14 of eth0 to CPU 1 Jul 22 02:30:15 NAS autotweak: Set CPU affinity of interrupt 206 and tx queue 15 of eth0 to CPU 2 Jul 22 02:30:15 NAS autotweak: Usable CPUs for eth1: 1-11,13-15 Jul 22 02:30:15 NAS autotweak: Set CPU affinity of interrupt 217 and tx queue 0 of eth1 to CPU 3 Jul 22 02:30:15 NAS autotweak: Set CPU affinity of interrupt 218 and tx queue 1 of eth1 to CPU 4 Jul 22 02:30:15 NAS autotweak: Set CPU affinity of interrupt 219 and tx queue 2 of eth1 to CPU 5 Jul 22 02:30:15 NAS autotweak: Set CPU affinity of interrupt 220 and tx queue 3 of eth1 to CPU 6 Jul 22 02:30:15 NAS autotweak: Set CPU affinity of interrupt 221 and tx queue 4 of eth1 to CPU 7 Jul 22 02:30:15 NAS autotweak: Set CPU affinity of interrupt 222 and tx queue 5 of eth1 to CPU 8 Jul 22 02:30:15 NAS autotweak: Set CPU affinity of interrupt 223 and tx queue 6 of eth1 to CPU 9 Jul 22 02:30:15 NAS autotweak: Set CPU affinity of interrupt 224 and tx queue 7 of eth1 to CPU 10 Jul 22 02:30:15 NAS autotweak: Set CPU affinity of interrupt 225 and tx queue 8 of eth1 to CPU 11 Jul 22 02:30:15 NAS autotweak: Set CPU affinity of interrupt 226 and tx queue 9 of eth1 to CPU 13 Jul 22 02:30:15 NAS autotweak: Set CPU affinity of interrupt 227 and tx queue 10 of eth1 to CPU 14 Jul 22 02:30:15 NAS autotweak: Set CPU affinity of interrupt 228 and tx queue 11 of eth1 to CPU 15 Jul 22 02:30:15 NAS autotweak: Set CPU affinity of interrupt 229 and tx queue 12 of eth1 to CPU 1 Jul 22 02:30:15 NAS autotweak: Set CPU affinity of interrupt 230 and tx queue 13 of eth1 to CPU 2 Jul 22 02:30:15 NAS autotweak: Set CPU affinity of interrupt 231 and tx queue 14 of eth1 to CPU 3 Jul 22 02:30:15 NAS autotweak: Set CPU affinity of interrupt 232 and tx queue 15 of eth1 to CPU 4 Jul 22 02:30:15 NAS autotweak: Maximum interface speed: 10000 Mb/s Jul 22 02:30:15 NAS autotweak: TCP parameters optimized

I received this email: Subject: cron for user root /usr/bin/run-parts /etc/cron.daily 1> /dev/null Body: error: error writing to /var/log/graphql-api.log.1: No space left on device error: error copying /var/log/graphql-api.log to /var/log/graphql-api.log.1: No space left on device Upon checking the /var/log folder, the graphql-api.log file was the largest (over 80MB). It seems most of it has to do with socket disconnections.

I just experienced something similar, but intead I got this email: error: error writing to /var/log/graphql-api.log.1: No space left on device error: error copying /var/log/graphql-api.log to /var/log/graphql-api.log.1: No space left on devic e

@JorgeB I'll be damned. It's running fine with one stick of RAM, and now I think I understand why: I think I mixed up the modules the last time I re-assembled this computer and they have different characteristics. Perhaps there is some incompatibility between them. One kit is single rank and the other kit is dual rank. There's some combination of the two kits that apparently allows it to work correctly under some circumstances (e.g. single rank in channel A and dual-rank in channel B, but not the other way around). I'll try re-adding a second matching stick later and seeing what happens. If that works, I'll see if I can find an exact match to one of the kits and order that. They're the same model of RAM and even use Micro dies, but one uses 16 modules and the other one just 8 modules. I had highly doubted that could be the problem because the server is absolutely rock-solid until write operations start happening to the parity-protected array and it made no sense that my memory configuration would be unstable, but not manifest itself in any other way in months of uptime. No errors while doing massive operations between pools and other shenanigans, all stress tests passing with flying colors, but a total freeze by simply writing something to the array. Anyway, I'll wait for the data rebuild to finish and try and debug this better. In any case, I'll mark this as resolved.

@itimpi and @JorgeB I have been fighting this again for about a week and I am 100% able to reproduce it. Additionally, one of my HDDs started to exhibit an increasing number of reallocated sectors, so I RMA'd it and the fact I now need to restore the array gave me some more insight into this. The machine freezes before it even writes anything to syslog or dmesg. I also have a NanoKVM connected to the machine and I ran tail -F /var/log/{syslog,dmesg} but the crash is so severe even the output to the screen contains absolutely nothing about any issue. Things that do not affect the crash: Plugins. Crash is 100% reproducible in safe mode. RAM. Memtest passes multiple times and the server is rock solid until the array comes into play. Read operations. As long as only read operations are performed, including parity checks and scrubbing, it's all fine. I did get one crash during a parity check, but I never got one again. Docker, VMs, network activity, system load, etc. I tried multiple combinations of everything I could think of, but even with everything here disabled, I still get the crash. Things that cause it to crash: Always: trying to restore the array with the new HDD I received Often: Mover and rsync'ing from the ZFS cache pool to /mnt/user0/. It's guaranteed to crash after a hundred gigabytes or so. Less often / not at all sometimes: rsync'ing from the ZFS cache pool directly to one of the disks (via /mnt/diskX) and/or moving data between the disks using the unbalance plugin. I'm really lost here because no logs contain anything useful and the syslog server (and the entire machine, apparently) dies before anything (assuming the crash produces logs) happens, but one thing is clear: the server is absolutely rock solid unless the parity-protected array is in the mix. Is there anything I can enable to produce additional logs?