So a bunch of docker containers installed, but it seems I can't start them...

 

A bunch of errors in the syslog attached. I have no idea how to interpret it.

tower-syslog-20241213-2131.zip

Try running memtest, or since memtest is only definitive if it finds errors, if you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.

I have not come as far as running a memtest yet. Bur lo and behold, suddenly the rig is running again, and I cantower-syslog-20241214-2132.zip actually open dockers. Emby for instance seems to work as normal, Plex gives me "Tower is currently unavailable", but this might be due to some IP conflicts, as my whole network has received new IP address since it was last running.

 

But for insurance, can anybody check my logs and see if it seems ok, or if there is something I definitively need to get done here?

There are still issues with the pool filesystem it just went read-only.

Yes you are right. I did a reboot after the diag file, and when it came back. Docker service did not start.

 

I will try to run a memtest and see if it finds anything. If memory is ok, I consider to start from scratch again. Is there a guide to help me on my way on this?

If memtest doesn't detect any errors, you can backup and re-format the pool, but still recommend this to continue testing

On 12/14/2024 at 8:55 AM, JorgeB said:

if you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.

 

56 minutes ago, JorgeB said:

If memtest doesn't detect any errors, you can backup and re-format the pool, but still recommend this to continue testing

 

So actually both my pool disks are now unmountable, I have no idea what happened there.

 

On memtest, I understand the following:

1. Run memtest on all 4 sticks

if fail:

2. Test each stick manually separately, and remove faulty ones

 

If setting this up again, do you recommend me to keep both Pool disks or have two identical ones, or even just the one on 1 TB?

 

Also how can I backup the pool disks when they are unmountable?

 

Run memtest first and then post new diags to see if the pool can still be repaired for backup.

Ok will do.

 

Which memtests do I need to do? I know you can do 24 hour tests and shorter tests. 

 

Also is there a way to fully remove Disk1 from Unraid? Even if I have physically removed it it still says USED/Free, or should I replace it with another disk? I do have one more of these 2 tb disks laying around, but I was initially planning on replacing it when I bought another new 20TB disk sometime in the future.

 

I am now in the process of backing up all my media, as I don't know if the array will work, if I end up with installing Unraid from scratch in the future.

 

Run the memtest from the boot menu.

 

If you men remove from the array without keeping the data, see here:

https://docs.unraid.net/legacy/FAQ/shrink-array/

Memtesting has started.

A whole HOST of errors. How can this happen from one moment to another? I did do memtest when I first set up the right in October of 2023. Moving on to testing individual ram.

 

I guess it is best to run it in with 1 CPU, not all.

So memtesting finished.

1 stick with 100s of errors.

1 stick with 1 single error, but popping up at the same address each run.

2 sticks with 0 errors.

 

Rebooted Unraid after removing the faulty sticks, and included the logs here. I guess its time to rebuild the docker image again?

PS: The cache pool started up now.

tower-diagnostics-20241215-2315.zip

18 hours ago, tormi said:

How can this happen from one moment to another?

Any piece of hardware can go bad at any time.

 

 

You should run a correcting scrub on the pool first

 

After an optimistic end to the evening last night, I started today with going up on a few dockers. Plex for instance worked perfectly, while sonarr and radar started but I could not access them.

Secondle the pool went offline, so I reproduced the steps you created for me the other day. I just finished scrubbing with the following result:

 

I check the syslog and get the following:

Dec 16 17:51:16 Tower kernel: BTRFS warning (device nvme0n1p1): checksum error at logical 31866134528 on dev /dev/nvme1n1p1, physical 698261504, root 5, inode 243945, offset 21573632, length 4096, links 1 (path: appdata/homepage/logs/homepage.log)
Dec 16 17:51:16 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 39, gen 0
Dec 16 17:51:20 Tower kernel: BTRFS warning (device nvme0n1p1): checksum error at logical 31866134528 on dev /dev/nvme0n1p1, physical 32948264960, root 5, inode 243945, offset 21573632, length 4096, links 1 (path: appdata/homepage/logs/homepage.log)
Dec 16 17:51:20 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 39, gen 0

 

So I guess there is still something wrong, so I proceeded to do a "correcting scrub", and I get the following:

Back to the uncorrectable errors:

Dec 16 17:56:44 Tower kernel: BTRFS warning (device nvme0n1p1): checksum error at logical 31866134528 on dev /dev/nvme1n1p1, physical 698261504, root 5, inode 243945, offset 21573632, length 4096, links 1 (path: appdata/homepage/logs/homepage.log)
Dec 16 17:56:44 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 40, gen 0
Dec 16 17:56:44 Tower kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 31866134528 on dev /dev/nvme1n1p1
Dec 16 17:56:47 Tower kernel: BTRFS warning (device nvme0n1p1): checksum error at logical 31866134528 on dev /dev/nvme0n1p1, physical 32948264960, root 5, inode 243945, offset 21573632, length 4096, links 1 (path: appdata/homepage/logs/homepage.log)
Dec 16 17:56:47 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 40, gen 0
Dec 16 17:56:47 Tower kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 31866134528 on dev /dev/nvme0n1p1

I do know that this "nvme0n1p1" is a somewhat older disk than the nvme1n1.

 

What to do next?

 

PS: I have also wondered if it at all makes sense to have a pool of 1 250GB NVME disk and 1 at 1TB. If something is wrong with the 250GB one, how should I proceeded to remove it from Unraid without loosing any data?

Edited December 16, 20241 yr by tormi

1 hour ago, tormi said:

What to do next?

Delete/restore the corrupt files and then re-run a scru to confirm 0 errors:

 

1 hour ago, tormi said:

I have also wondered if it at all makes sense to have a pool of 1 250GB NVME disk and 1 at 1TB. If something is wrong with the 250GB one, how should I proceeded to remove it from Unraid without loosing any data?

As long as you know that only 250GB are usable, it's an option to have redundancy.

I finally understood what file to look for  The result now:

UUID: 1fd1ba22-3954-4321-82d8-d7dac32a05c6 Scrub started: Mon Dec 16 19:19:49 2024 Status: finished Duration: 0:00:22 Total to scrub: 65.81GiB Rate: 2.99GiB/s Error summary: no errors found on the scrub. I still have this in the syslog though:

Dec 16 19:19:05 Tower kernel: BTRFS warning (device nvme0n1p1): checksum error at logical 31866134528 on dev /dev/nvme1n1p1, physical 698261504, root 5, inode 243945, offset 21573632: path resolving failed with ret=1
Dec 16 19:19:05 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 41, gen 0
Dec 16 19:19:08 Tower kernel: BTRFS warning (device nvme0n1p1): checksum error at logical 31866134528 on dev /dev/nvme0n1p1, physical 32948264960, root 5, inode 243945, offset 21573632: path resolving failed with ret=1
Dec 16 19:19:08 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 41, gen 0

 

Does this mean it is all good now? Or do I need other steps?

 

In regards to the 250GB, do I understand you correctly that my setup is good and allows redundancy the way it is now, as long as I don't exceed 250GB storage in cache?

 

 

Edited December 16, 20241 yr by tormi

11 minutes ago, tormi said:

In regards to the 250GB, do I understand you correctly that my setup is good and allows redundancy the way it is now, as long as I don't exceed 250GB storage in cache?

Correct.

 

See below for how to reset the btrfs stats and better pool monitoring:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/#findComment-700582

 

I don't know if you answered about these errors: 

"Dec 16 19:19:05 Tower kernel: BTRFS warning (device nvme0n1p1): checksum error at logical 31866134528 on dev /dev/nvme1n1p1, physical 698261504, root 5, inode 243945, offset 21573632: path resolving failed with ret=1 Dec 16 19:19:05 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 41, gen 0 Dec 16 19:19:08 Tower kernel: BTRFS warning (device nvme0n1p1): checksum error at logical 31866134528 on dev /dev/nvme0n1p1, physical 32948264960, root 5, inode 243945, offset 21573632: path resolving failed with ret=1 Dec 16 19:19:08 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 41, gen 0"

 

Should I/can I do anything about this, or should I now just try to use the setup as "normal"? 

Try rebooting to see if those go away.

Damn!

 

I was so optimistic now. I rebooted, errors went away but as I rebuilt the docker file and started a few dockers I first got the "Excecution error" "Server error", and checking the syslog I got the following:

ec 16 21:30:14 Tower kernel: BTRFS: error (device nvme0n1p1: state A) in __btrfs_free_extent:3072: errno=-2 No such entry
Dec 16 21:30:14 Tower kernel: BTRFS info (device nvme0n1p1: state EA): forced readonly
Dec 16 21:30:14 Tower kernel: BTRFS error (device nvme0n1p1: state EA): failed to run delayed ref for logical 33457631232 num_bytes 12288 type 178 action 2 ref_mod 1: -2
Dec 16 21:30:14 Tower kernel: BTRFS: error (device nvme0n1p1: state EA) in btrfs_run_delayed_refs:2149: errno=-2 No such entry
Dec 16 21:30:15 Tower kernel: docker0: port 4(vethafa210b) entered blocking state
Dec 16 21:30:15 Tower kernel: docker0: port 4(vethafa210b) entered disabled state
Dec 16 21:30:15 Tower kernel: device vethafa210b entered promiscuous mode
Dec 16 21:30:15 Tower kernel: docker0: port 4(vethafa210b) entered blocking state
Dec 16 21:30:15 Tower kernel: docker0: port 4(vethafa210b) entered forwarding state
Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 7968997376, length 4096.
Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 7578939392, length 4096.
Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 7968993280, length 4096.
Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 15564448 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 2
Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 14802616 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 15564440 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
Dec 16 21:30:15 Tower kernel: docker0: port 4(vethafa210b) entered disabled state
Dec 16 21:30:15 Tower kernel: docker0: port 4(vethafa210b) entered disabled state
Dec 16 21:30:15 Tower kernel: device vethafa210b left promiscuous mode
Dec 16 21:30:15 Tower kernel: docker0: port 4(vethafa210b) entered disabled state
Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 505610240, length 4096.
Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 237174784, length 4096.
Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 505593856, length 4096.
Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 237158400, length 4096.
Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 987520 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 2
Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 463232 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 2
Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 987488 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 2
Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 463200 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 2
Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 227229696, length 4096.
Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 495665152, length 4096.
Dec 16 21:30:15 Tower kernel: loop: Write error at byte offset 227278848, length 4096.
Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 443808 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 2
Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 968096 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 2
Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
Dec 16 21:30:15 Tower kernel: I/O error, dev loop2, sector 443904 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 2
Dec 16 21:30:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
Dec 16 21:30:16 Tower kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2494: errno=-5 IO failure (Error while writing out transaction)
Dec 16 21:30:16 Tower kernel: BTRFS info (device loop2: state E): forced readonly
Dec 16 21:30:16 Tower kernel: BTRFS warning (device loop2: state E): Skipping commit of aborted transaction.
Dec 16 21:30:16 Tower kernel: BTRFS: error (device loop2: state EA) in cleanup_transaction:1992: errno=-5 IO failure

 

And on reboot, the cache disks are unmountable again.

PS: I do have another 500GB Samsung 970 EVOPlus drive, with no errors as far as I can see. Should I try to switch it out with the 250? Or is this prematurely?

Edited December 17, 20241 yr by tormi

Pool filesystem has other issues, recommend backing up the pool and recreating it.

Hmm. 

 

How do I backup the pool? Just copy of the data?

How do I recreate it? I guess I can Google that.

 

Should I wait with replacing the 250GB drive until I have done the two steps above?

Yes, just copy the data, then re-format.

 

You can replace the device after the backup and before re-formatiing.

So it seems things are starting to work again now. Installed the new 500gb harddrive in pool, and the first thing I noticed is that is is 30 degrees cooler than the last one.

 

A final checkup (I hope):

1. Is these shares ok or would you suggest other?

2. Logfile included, does it look good? No more red flags in the syslog by the way.

tower-diagnostics-20241217-1418.zip

Syslog looks OK for now, as for the shares, default shares could probably be all set to cache=only, but first you need to confirm if they have data on the array, and if yes, move it first.