tormi

Damn! I was so optimistic now. I rebooted, errors went away but as I rebuilt the docker file and started a few dockers I first got the "Excecution error" "Server error", and checking the syslog I got the following: ec 16 21:30:14 Tower kernel: BTRFS: error (device nvme0n1p1: state A) in __btrfs_free_extent:3072: errno=-2 No such entry Dec 16 21:30:14 Tower kernel: BTRFS info (device nvme0n1p1: state EA): forced readonly Dec 16 21:30:14 Tower kernel: BTRFS error (device nvme0n1p1: state EA): failed to run delayed ref for logical 33457631232 num_bytes 12288 type 178 action 2 ref_mod 1: -2 Dec 16 21:30:14 Tower kernel: BTRFS: error (device nvme0n1p1: state EA) in btrfs_run_delayed_refs:2149: errno=-2 No such entry Dec 16 21:30:15 Tower kernel: docker0: port 4(vethafa210b) entered blocking state Dec 16 21:30:15 Tower kernel: docker0: port 4(vethafa210b) entered disabled state Dec 16 21:30:15 Tower kernel: device vethafa210b entered promiscuous mode Dec 16 21:30:15 Tower kernel: docker0: port 4(vethafa210b) entered blocking state Dec 16 21:30:15 Tower kernel: docker0: port 4(vethafa210b) entered forwarding state Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 7968997376, length 4096. Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 7578939392, length 4096. Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 7968993280, length 4096. Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 15564448 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 2 Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0 Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 14802616 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2 Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0 Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 15564440 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2 Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0 Dec 16 21:30:15 Tower kernel: docker0: port 4(vethafa210b) entered disabled state Dec 16 21:30:15 Tower kernel: docker0: port 4(vethafa210b) entered disabled state Dec 16 21:30:15 Tower kernel: device vethafa210b left promiscuous mode Dec 16 21:30:15 Tower kernel: docker0: port 4(vethafa210b) entered disabled state Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 505610240, length 4096. Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 237174784, length 4096. Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 505593856, length 4096. Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 237158400, length 4096. Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 987520 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 2 Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 463232 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 2 Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0 Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0 Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 987488 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 2 Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 463200 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 2 Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0 Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0 Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 227229696, length 4096. Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 495665152, length 4096. Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 227278848, length 4096. Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 443808 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 2 Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0 Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 968096 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 2 Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0 Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 443904 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 2 Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0 Dec 16 21:30:16 Tower kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2494: errno=-5 IO failure (Error while writing out transaction) Dec 16 21:30:16 Tower kernel: BTRFS info (device loop2: state E): forced readonly Dec 16 21:30:16 Tower kernel: BTRFS warning (device loop2: state E): Skipping commit of aborted transaction. Dec 16 21:30:16 Tower kernel: BTRFS: error (device loop2: state EA) in cleanup_transaction:1992: errno=-5 IO failure And on reboot, the cache disks are unmountable again. PS: I do have another 500GB Samsung 970 EVOPlus drive, with no errors as far as I can see. Should I try to switch it out with the 250? Or is this prematurely?

I don't know if you answered about these errors: "Dec 16 19:19:05 Tower kernel: BTRFS warning (device nvme0n1p1): checksum error at logical 31866134528 on dev /dev/nvme1n1p1, physical 698261504, root 5, inode 243945, offset 21573632: path resolving failed with ret=1 Dec 16 19:19:05 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 41, gen 0 Dec 16 19:19:08 Tower kernel: BTRFS warning (device nvme0n1p1): checksum error at logical 31866134528 on dev /dev/nvme0n1p1, physical 32948264960, root 5, inode 243945, offset 21573632: path resolving failed with ret=1 Dec 16 19:19:08 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 41, gen 0" Should I/can I do anything about this, or should I now just try to use the setup as "normal"?

I finally understood what file to look for The result now: UUID: 1fd1ba22-3954-4321-82d8-d7dac32a05c6 Scrub started: Mon Dec 16 19:19:49 2024 Status: finished Duration: 0:00:22 Total to scrub: 65.81GiB Rate: 2.99GiB/s Error summary: no errors found on the scrub. I still have this in the syslog though: Dec 16 19:19:05 Tower kernel: BTRFS warning (device nvme0n1p1): checksum error at logical 31866134528 on dev /dev/nvme1n1p1, physical 698261504, root 5, inode 243945, offset 21573632: path resolving failed with ret=1 Dec 16 19:19:05 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 41, gen 0 Dec 16 19:19:08 Tower kernel: BTRFS warning (device nvme0n1p1): checksum error at logical 31866134528 on dev /dev/nvme0n1p1, physical 32948264960, root 5, inode 243945, offset 21573632: path resolving failed with ret=1 Dec 16 19:19:08 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 41, gen 0 Does this mean it is all good now? Or do I need other steps? In regards to the 250GB, do I understand you correctly that my setup is good and allows redundancy the way it is now, as long as I don't exceed 250GB storage in cache?

After an optimistic end to the evening last night, I started today with going up on a few dockers. Plex for instance worked perfectly, while sonarr and radar started but I could not access them. Secondle the pool went offline, so I reproduced the steps you created for me the other day. I just finished scrubbing with the following result: I check the syslog and get the following: Dec 16 17:51:16 Tower kernel: BTRFS warning (device nvme0n1p1): checksum error at logical 31866134528 on dev /dev/nvme1n1p1, physical 698261504, root 5, inode 243945, offset 21573632, length 4096, links 1 (path: appdata/homepage/logs/homepage.log) Dec 16 17:51:16 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 39, gen 0 Dec 16 17:51:20 Tower kernel: BTRFS warning (device nvme0n1p1): checksum error at logical 31866134528 on dev /dev/nvme0n1p1, physical 32948264960, root 5, inode 243945, offset 21573632, length 4096, links 1 (path: appdata/homepage/logs/homepage.log) Dec 16 17:51:20 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 39, gen 0 So I guess there is still something wrong, so I proceeded to do a "correcting scrub", and I get the following: Back to the uncorrectable errors: Dec 16 17:56:44 Tower kernel: BTRFS warning (device nvme0n1p1): checksum error at logical 31866134528 on dev /dev/nvme1n1p1, physical 698261504, root 5, inode 243945, offset 21573632, length 4096, links 1 (path: appdata/homepage/logs/homepage.log) Dec 16 17:56:44 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 40, gen 0 Dec 16 17:56:44 Tower kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 31866134528 on dev /dev/nvme1n1p1 Dec 16 17:56:47 Tower kernel: BTRFS warning (device nvme0n1p1): checksum error at logical 31866134528 on dev /dev/nvme0n1p1, physical 32948264960, root 5, inode 243945, offset 21573632, length 4096, links 1 (path: appdata/homepage/logs/homepage.log) Dec 16 17:56:47 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 40, gen 0 Dec 16 17:56:47 Tower kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 31866134528 on dev /dev/nvme0n1p1 I do know that this "nvme0n1p1" is a somewhat older disk than the nvme1n1. What to do next? PS: I have also wondered if it at all makes sense to have a pool of 1 250GB NVME disk and 1 at 1TB. If something is wrong with the 250GB one, how should I proceeded to remove it from Unraid without loosing any data?

So memtesting finished. 1 stick with 100s of errors. 1 stick with 1 single error, but popping up at the same address each run. 2 sticks with 0 errors. Rebooted Unraid after removing the faulty sticks, and included the logs here. I guess its time to rebuild the docker image again? PS: The cache pool started up now. tower-diagnostics-20241215-2315.zip

Memtesting has started. A whole HOST of errors. How can this happen from one moment to another? I did do memtest when I first set up the right in October of 2023. Moving on to testing individual ram. I guess it is best to run it in with 1 CPU, not all.

Ok will do. Which memtests do I need to do? I know you can do 24 hour tests and shorter tests. Also is there a way to fully remove Disk1 from Unraid? Even if I have physically removed it it still says USED/Free, or should I replace it with another disk? I do have one more of these 2 tb disks laying around, but I was initially planning on replacing it when I bought another new 20TB disk sometime in the future. I am now in the process of backing up all my media, as I don't know if the array will work, if I end up with installing Unraid from scratch in the future.

So actually both my pool disks are now unmountable, I have no idea what happened there. On memtest, I understand the following: 1. Run memtest on all 4 sticks if fail: 2. Test each stick manually separately, and remove faulty ones If setting this up again, do you recommend me to keep both Pool disks or have two identical ones, or even just the one on 1 TB? Also how can I backup the pool disks when they are unmountable?

Yes you are right. I did a reboot after the diag file, and when it came back. Docker service did not start. I will try to run a memtest and see if it finds anything. If memory is ok, I consider to start from scratch again. Is there a guide to help me on my way on this?

I have not come as far as running a memtest yet. Bur lo and behold, suddenly the rig is running again, and I cantower-syslog-20241214-2132.zip actually open dockers. Emby for instance seems to work as normal, Plex gives me "Tower is currently unavailable", but this might be due to some IP conflicts, as my whole network has received new IP address since it was last running. But for insurance, can anybody check my logs and see if it seems ok, or if there is something I definitively need to get done here?

So a bunch of docker containers installed, but it seems I can't start them... A bunch of errors in the syslog attached. I have no idea how to interpret it. tower-syslog-20241213-2131.zip

So i have gone forward with the process. A bunch of the dockers/apps have been installed from "previous apps", but it seems it is stuck at my "TandoorReciepes" app. It has been "pulling fs layer" for at least 1 hour now on this app... Just leave or do something else?

So all I do is click the "move" button at the "Main" tab, then I set the "System" array to cache only. After this I rebuild the docker container if needed? Do I need to make any path alterations in any other places? This is the settings for the "system" data btw: Should I not include a secondary storage?

I will certainly try this. Just one question before I continue. In the docker settings this is what it looks like now, I put no on "enable docker". Notice the vDisk location. I find the docker file at 22,5GB in disk2/system/docker which is a array disk. Is the directory location the same in the settings? When I choose the path manually I get to /mnt/disk2/system/docker/docker.img Just wondering if the path somehow has got distorted? Should I try to point it directly to that path myself? If so can you verify that the appdata storage folder is correct? Also I notice the Docker vDisk size is set to 20GB while mine is actually 21,5... Could this be an issue?

Latest diagtower-diagnostics-20241212-2007.zip

This is my Docker settings:

I did update them, but it says "not available". Still at least I can see the NUT plugin is working...

Hmm, I restarted the server again, now the NUT Tool is working, and the errors are gone. I send you the syslog again in PM. Still no dockers start though. There are plugins that need an update, but I don't know if it is safe to install plugins before dockers is up and running:

Checked the syslog. No referece to corrupt files but this is repeating: Dec 12 04:44:29 Tower usbhid-ups[2632]: libusb1: Could not open any HID devices: insufficient permissions on everything Dec 12 04:44:31 Tower upsmon[2640]: Poll UPS [[email protected]] failed - Data stale Dec 12 04:44:31 Tower usbhid-ups[2632]: libusb1: Could not open any HID devices: insufficient permissions on everything Dec 12 04:44:33 Tower usbhid-ups[2632]: libusb1: Could not open any HID devices: insufficient permissions on everything Dec 12 04:44:35 Tower usbhid-ups[2632]: libusb1: Could not open any HID devices: insufficient permissions on everything Dec 12 04:44:36 Tower upsmon[2640]: Poll UPS [[email protected]] failed - Data stale Dec 12 04:44:37 Tower usbhid-ups[2632]: libusb1: Could not open any HID devices: insufficient permissions on everything Dec 12 04:44:39 Tower usbhid-ups[2632]: libusb1: Could not open any HID devices: insufficient permissions on everything Dec 12 04:44:41 Tower upsmon[2640]: Poll UPS [[email protected]] failed - Data stale Dec 12 04:44:41 Tower usbhid-ups[2632]: libusb1: Could not open any HID devices: insufficient permissions on everything Dec 12 04:44:43 Tower usbhid-ups[2632]: libusb1: Could not open any HID devices: insufficient permissions on everything Dec 12 04:44:45 Tower usbhid-ups[2632]: libusb1: Could not open any HID devices: insufficient permissions on everything Dec 12 04:44:46 Tower upsmon[2640]: Poll UPS [[email protected]] failed - Data stale Dec 12 04:44:47 Tower usbhid-ups[2632]: libusb1: Could not open any HID devices: insufficient permissions on everything Dec 12 04:44:49 Tower usbhid-ups[2632]: libusb1: Could not open any HID devices: insufficient permissions on everything Dec 12 04:44:51 Tower upsmon[2640]: Poll UPS [[email protected]] failed - Data stale Dec 12 04:44:51 Tower usbhid-ups[2632]: libusb1: Could not open any HID devices: insufficient permissions on everything Dec 12 04:44:53 Tower usbhid-ups[2632]: libusb1: Could not open any HID devices: insufficient permissions on everything Dec 12 04:44:55 Tower usbhid-ups[2632]: libusb1: Could not open any HID devices: insufficient permissions on everything Dec 12 04:44:56 Tower upsmon[2640]: Poll UPS [[email protected]] failed - Data stale Dec 12 04:44:57 Tower usbhid-ups[2632]: libusb1: Could not open any HID devices: insufficient permissions on everything Dec 12 04:44:59 Tower usbhid-ups[2632]: libusb1: Could not open any HID devices: insufficient permissions on everything Dec 12 04:44:59 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update Considering removing the Network UPS tool for UNRAID, but don't know if that would fix anything. (I am scared of doing changes.)

Seem like something is "Uncorrectable"... I did a reboot after and took a new diag report. UUID: 1fd1ba22-3954-4321-82d8-d7dac32a05c6 Scrub started: Wed Dec 11 11:21:57 2024 Status: finished Duration: 0:00:09 Total to scrub: 23.88GiB Rate: 2.65GiB/s Error summary: csum=2 Corrected: 0 Uncorrectable: 2 Unverified: 0 tower-diagnostics.zip

Should I remove thoose cables to, or can I just leave them now that I removed the drive? Scrub Result (I did not choose to correct anything) UUID: 1fd1ba22-3954-4321-82d8-d7dac32a05c6 Scrub started: Wed Dec 11 04:59:24 2024 Status: finished Duration: 0:00:11 Total to scrub: 23.88GiB Rate: 2.17GiB/s Error summary: csum=2 Corrected: 0 Uncorrectable: 0 Unverified: 0

Hi @JorgeB. That 2tb disk had serious issues a while ago. I thought I had removed it from Unraid as I don't need that disk anymore. Could you really see that the issue was cables? In that case I will throw away the cables if I put in a new disk. Anyways, the disk is now removed, not the cables I expect that does not matter? tower-diagnostics-20241210-2009.zip

tower-diagnostics-20241210-1323.zipSeems like something happened in a positive way, but I can see Docker service "failed to start".

tower-diagnostics-20241210-1047.zipHere they are.

Hi. I am still a noob. I shut down my Unraid server end of July, because of houserenovations, and fired it up again today, and I got shot in the face by the following. I am sorry that I really don't understand how this happened at all. Only think I can think of is that I moved the server physically from one location in the house to another. But I have not touched anything, and it seems everything is alright on the inside. Can anybody point me in the right direction to help myself here? PS: Both cache disks are quite new, before this happened, as I setup Unraid fall of -23.