I've been having tons of struggles with my server or months now, but it's never been bad enough for me to want to come here and post about it.

- After immediate startup, it almost always reports an unclean shutdown from before, usually because it gets stuck trying to unmount a disk and has to wait for the timeout. Even when all services are shutdown and all containers are stopped, it refuses to unmount one of the disks, disk3 I think.

- Docker can run for anywhere between 2 days and 3 weeks before it'll suddenly start having issues, usually a container or two will stop, most will keep running, but the Docker service tab completely stops functioning, reporting 'Docker service could not be started'. If it's still available, and I try to start a container, it'll report "403 error".

- Every now and then, when it comes to this, I'll end up stopping the service manually, deleting the vDisk image file, and restarting, and it's magically fixed.

 

This latest issue is a bit new because almost every container was stopped suddenly, and I noticed for the past 2 days (that's as far back as my remote log retention goes in the free tier), there have been around 1400 log lines being reported every hour.

The start of this image is what it should look like, maybe only 20 lines per hour. Usually only blue (INFO) lines, too.

 

I don't understand what's going wrong with it now, there are lots of scary words like corruption and checksum fails. It often has reported issues with /dev/loop2, but I don't fully understand the what/why.

 

I'm just getting tired of this whole thing, because all the drives are still pretty new (February 2024) and my server hasn't had a good long uptime streak in quite a while.

tower-diagnostics-20250308-1123.zip

Your appdata, domains, and system shares have files on the array. Ideally, these Docker/VM related shares would have all files on cache or other pool, with nothing on the array, so Docker/VM will perform better, and so array disks can spin down since these files are always open.

 

 More concerning, lots of these in syslog

Mar  5 07:01:35 Tower kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 12153626 off 754589696 csum 0x829b5324 expected csum 0xcba57e8a mirror 1
...
Mar  7 23:01:03 Tower kernel: BTRFS warning (device loop2): csum failed root 5 ino 2511644 off 1069056 csum 0x174d5f06 expected csum 0x7a5f5942 mirror 1

btrfs csum errors are usually bad RAM.

 

You shouldn't attempt to run any computer unless RAM is working perfectly. Everything goes through RAM, the OS and other executable code, YOUR DATA, everything. The CPU can't do anything with anything until it is loaded into RAM.

 

memtest is on the boot menu.

1 hour ago, trurl said:

You shouldn't attempt to run any computer unless RAM is working perfectly. Everything goes through RAM, the OS and other executable code, YOUR DATA, everything. The CPU can't do anything with anything until it is loaded into RAM.

 

memtest is on the boot menu.

It's news to me that this would be a RAM issue, but sure; I can't easily run the full memtest suite, but I assumed that with the array stopped and running 87% of my memory against memtest (the live testing plugin in Unraid CA), I'd catch something - but I'm over an hour into the test and nothing's errored. I'm currently on the Walking Ones test, which I think might be the last, or at least very close to. If it hasn't errored yet, I doubt it will error after all these tests.

 

I know that's not foolproof, there's a fair chance that the remaining 13% of RAM that I can't test will contain problematic sectors of RAM, but still... I'm using both of the two monitors I have, and my server is an inconvenient location to be running memtest on from pure boot.

1 hour ago, trurl said:

More concerning, lots of these in syslog

Mar  5 07:01:35 Tower kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 12153626 off 754589696 csum 0x829b5324 expected csum 0xcba57e8a mirror 1
...
Mar  7 23:01:03 Tower kernel: BTRFS warning (device loop2): csum failed root 5 ino 2511644 off 1069056 csum 0x174d5f06 expected csum 0x7a5f5942 mirror 1

btrfs csum errors are usually bad RAM.

When I first saw these and looked them up, I could use find /mnt/cache -inum <ino> to identify the file with corruption (then I just deleted them, it was some movie file I didn't really care about), but currently, it doesn't do that. I'm unsure if it's because inodes move around during array start/stop or something else (I ran it while the array was started ofc).

 

The only real conceptual idea I have is that some container is spamming my Docker vDisk file like crazy, and inflating it. It feels like Docker is close to whatever problem my server is having, and deleting vDisk has continued to solve my problems for

reasons I'll never understand. That said, I don't know to confirm my suspicion, and don't have real proof.

 

1 hour ago, trurl said:

Your appdata, domains, and system shares have files on the array. Ideally, these Docker/VM related shares would have all files on cache or other pool, with nothing on the array, so Docker/VM will perform better, and so array disks can spin down since these files are always open.

I don't actually know how this has happened, nor really how I'd fix it. I keep having issues or weirdness with the Mover application - it's so janky. A lot of Unraid is like that imo - janky. I suspect I can just literally move them, but where to? I'm not super confident.

Edited March 8, 20251 yr by Xevion

Nevermind, it finally found something (Walking Zeroes); although I don't know what to do from here though.

 

I can only assume this means replacing a stick, possibly multiple sticks. I run 4 x 8 GB DDR4 3200 MHz, just some old stuff from a gaming computer, even has RGB lol.

I'm gonna assume I have to do a binary search to find which stick (hoping it's just one) failed, and replace it.

 

Besides that though, I'm unsure whether to replace all of it, or just find a single 8GB stick. Really would prefer to go for 64GB total, but maybe it's unnecessary for my little server.