so i have a list of seemingly unrelated issues that i cant find the cause of any of them:
- server randomly restarts (this one doesn't happen often but most of the time happens overnight)
- dockers are having issues reading there configs?.. some dockers are crashing in there logs saying they cant read there configs or something along those lines cant read file: x but not all dockers only a few at a time (rebooting fixes this for a few hours then a diffrent docker would have a issue Like this)
- docker containers just not starting and when i open logs they close strait away (rebooting fixes this)
- this has only happened once so not sure if related but i couldn't access any shares for like 30 mins and while i was trying to figure out why it suddenly started working again in windows
- the dashboard being completely unresponsive until i rebooted server then worked fine again (has only happened a few times)

all of this only started about 3 weeks ago after i replaced the parity drive then added a new drive to the array

i have attached 2 digs one from a few weeks ago and one from today

any help is greatly applicated i been trying to figure this out but i can and I'm sick of having to restart the server almost daily

friday-diagnostics-20251011-1617-todays.zipfriday-diagnostics-20251003-1040-a-few-days-old.zip

Server restarting by itself is almost always a hardware issue, and that could cause other issues, make sure this has been taken care of:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/#findComment-819173

If it doesn't help, and since you have multiple RAM sticks, try using the server with just one pair, if the same try with the other one, that will basically rule out bad RAM.

so i am looking in to what you mentioned above

10 hours ago, JorgeB said:

Server restarting by itself is almost always a hardware issue, and that could cause other issues, make sure this has been taken care of:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/#findComment-819173

If it doesn't help, and since you have multiple RAM sticks, try using the server with just one pair, if the same try with the other one, that will basically rule out bad RAM.

and this morning i found the server with every docker not either stopped or started just not working all logs wouldn't open and no dockers where working i saved digs before rebooting then everything started and was working again just fine. the digs i saved:

friday-diagnostics-20251012-0833.zip

Logs show issues with the pool, and as a consequence of that, the Docker image, syslog rotated, so I cannot see the start of the problem, post new diags after the reboot to check current pool status.

new digs 4 hours after reboot:
friday-diagnostics-20251012-1243.zip

all my pools look ok

have am now getting this from docker

friday-diagnostics-20251012-2021.zip

Oct 12 08:37:16 FRIDAY kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 31, gen 0

Btrfs is detecting data corruption on the pool, scrub it and post the results.

it seems its now unmountable not sure whys that happened


i did a btrfs check and got this
[1/8] checking log

[2/8] checking root items

[3/8] checking extents

parent transid verify failed on 4976128163840 wanted 549757157815 found 1343927

parent transid verify failed on 4976128163840 wanted 549757157815 found 1343927

parent transid verify failed on 4976128163840 wanted 549757157815 found 1343927

Ignoring transid failure

[4/8] checking free space tree

[5/8] checking fs roots

parent transid verify failed on 4976128163840 wanted 549757157815 found 1343927

Ignoring transid failure

Wrong generation of child node/leaf, wanted: 1343927, have: 549757157815

root 5 inode 260 errors 400, nbytes wrong

root 5 inode 261 errors 2001, no inode item, link count wrong

unresolved ref dir 257 index 3 namelen 7 name libvirt filetype 2 errors 4, no inode ref

root 5 inode 3832501 errors 2001, no inode item, link count wrong

unresolved ref dir 256 index 32 namelen 5 name media filetype 2 errors 4, no inode ref

root 5 inode 16821099 errors 2001, no inode item, link count wrong

unresolved ref dir 256 index 41 namelen 3 name ssl filetype 2 errors 4, no inode ref

root 5 inode 16821102 errors 2001, no inode item, link count wrong

unresolved ref dir 256 index 42 namelen 7 name .config filetype 2 errors 4, no inode ref

root 5 inode 16821106 errors 2001, no inode item, link count wrong

unresolved ref dir 256 index 43 namelen 4 name .XDG filetype 2 errors 4, no inode ref

root 5 inode 16821107 errors 2001, no inode item, link count wrong

unresolved ref dir 256 index 44 namelen 6 name .local filetype 2 errors 4, no inode ref

root 5 inode 16821114 errors 2001, no inode item, link count wrong

unresolved ref dir 256 index 45 namelen 7 name .bashrc filetype 1 errors 4, no inode ref

root 5 inode 38166307 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1517 namelen 21 name binhex-readarr-spoken filetype 2 errors 4, no inode ref

root 5 inode 38363408 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1538 namelen 17 name Plex-Media-Server filetype 2 errors 4, no inode ref

root 5 inode 38412160 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1542 namelen 21 name binhex-readarr-ebooks filetype 2 errors 4, no inode ref

root 5 inode 38520127 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1549 namelen 16 name binhex-overseerr filetype 2 errors 4, no inode ref

root 5 inode 38635505 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1559 namelen 7 name mariadb filetype 2 errors 4, no inode ref

root 5 inode 38636014 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1562 namelen 17 name PostgreSQL_Immich filetype 2 errors 4, no inode ref

root 5 inode 39258918 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1597 namelen 5 name nginx filetype 2 errors 4, no inode ref

root 5 inode 40431862 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1653 namelen 8 name tautulli filetype 2 errors 4, no inode ref

root 5 inode 41326616 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1689 namelen 13 name binhex-radarr filetype 2 errors 4, no inode ref

root 5 inode 41832557 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1714 namelen 17 name mysql-pterodactyl filetype 2 errors 4, no inode ref

root 5 inode 41832693 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1718 namelen 15 name binhex-krusader filetype 2 errors 4, no inode ref

root 5 inode 42180273 errors 2001, no inode item, link count wrong

unresolved ref dir 256 index 49 namelen 16 name pterodactyl-node filetype 2 errors 4, no inode ref

root 5 inode 42337249 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1751 namelen 16 name binhex-delugevpn filetype 2 errors 4, no inode ref

root 5 inode 42680375 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1765 namelen 15 name binhex-prowlarr filetype 2 errors 4, no inode ref

root 5 inode 44046758 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1767 namelen 28 name Nginx-Proxy-Manager-Official filetype 2 errors 4, no inode ref

root 5 inode 44094881 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1773 namelen 15 name binhex-jellyfin filetype 2 errors 4, no inode ref

root 5 inode 44234479 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1774 namelen 20 name postgresql17-finance filetype 2 errors 4, no inode ref

root 5 inode 44234482 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1775 namelen 12 name postgresql16 filetype 2 errors 4, no inode ref

root 5 inode 44234487 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1776 namelen 21 name postgresql17-synoptic filetype 2 errors 4, no inode ref

root 5 inode 44234516 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1777 namelen 8 name registry filetype 2 errors 4, no inode ref

root 5 inode 44239286 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1778 namelen 8 name crafty-4 filetype 2 errors 4, no inode ref

root 5 inode 44240605 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1779 namelen 13 name binhex-sonarr filetype 2 errors 4, no inode ref

root 5 inode 44245623 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1780 namelen 6 name immich filetype 2 errors 4, no inode ref

root 5 inode 44247863 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1781 namelen 6 name onedev filetype 2 errors 4, no inode ref

root 5 inode 44249963 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1782 namelen 14 name audiobookshelf filetype 2 errors 4, no inode ref

root 5 inode 44251139 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1783 namelen 12 name postgresql15 filetype 2 errors 4, no inode ref

root 5 inode 44254052 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1784 namelen 13 name binhex-lidarr filetype 2 errors 4, no inode ref

root 5 inode 44257352 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1785 namelen 16 name postgresql17-dbs filetype 2 errors 4, no inode ref

root 5 inode 44258667 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1786 namelen 21 name postgresql16-imperium filetype 2 errors 4, no inode ref

root 5 inode 44260106 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1787 namelen 10 name watchstate filetype 2 errors 4, no inode ref

root 5 inode 44260233 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1788 namelen 14 name postgresql15-0 filetype 2 errors 4, no inode ref

root 5 inode 44261589 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1789 namelen 12 name postgresql17 filetype 2 errors 4, no inode ref

root 5 inode 44262944 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1790 namelen 20 name postgresql16-persnal filetype 2 errors 4, no inode ref

root 5 inode 44265570 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1791 namelen 13 name postgresql153 filetype 2 errors 4, no inode ref

root 5 inode 44266865 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1792 namelen 13 name postgresql151 filetype 2 errors 4, no inode ref

root 5 inode 44268160 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1793 namelen 23 name postgresql17-mattermost filetype 2 errors 4, no inode ref

root 5 inode 44269566 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1794 namelen 11 name pterodactyl filetype 2 errors 4, no inode ref

root 5 inode 44269603 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1795 namelen 7 name trilium filetype 2 errors 4, no inode ref

root 5 inode 44269750 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1796 namelen 8 name dizquetv filetype 2 errors 4, no inode ref

root 5 inode 44270355 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1797 namelen 8 name heimdall filetype 2 errors 4, no inode ref

root 5 inode 44270384 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1798 namelen 6 name axigen filetype 2 errors 4, no inode ref

root 5 inode 44270506 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1799 namelen 7 name databag filetype 2 errors 4, no inode ref

root 5 inode 44270510 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1800 namelen 19 name Plex-Auto-Languages filetype 2 errors 4, no inode ref

root 5 inode 44270513 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1801 namelen 8 name Manyfold filetype 2 errors 4, no inode ref

root 5 inode 44270515 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1802 namelen 11 name cloudbeaver filetype 2 errors 4, no inode ref

root 5 inode 44270534 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1803 namelen 8 name goaccess filetype 2 errors 4, no inode ref

root 5 inode 44270540 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1804 namelen 5 name PuTTY filetype 2 errors 4, no inode ref

root 5 inode 44270585 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1805 namelen 6 name homarr filetype 2 errors 4, no inode ref

root 5 inode 44270591 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1806 namelen 20 name HomeAssistant_inabox filetype 2 errors 4, no inode ref

root 5 inode 44270598 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1807 namelen 11 name snippet-box filetype 2 errors 4, no inode ref

root 5 inode 44270600 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1808 namelen 7 name readarr filetype 2 errors 4, no inode ref

root 5 inode 44270602 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1809 namelen 6 name tunarr filetype 2 errors 4, no inode ref

root 5 inode 44270603 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1810 namelen 8 name playitgg filetype 2 errors 4, no inode ref

root 5 inode 44270604 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1811 namelen 9 name gitlab-ce filetype 2 errors 4, no inode ref

root 5 inode 44270605 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1812 namelen 8 name local_ai filetype 2 errors 4, no inode ref

root 5 inode 44270606 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1813 namelen 10 name mattermost filetype 2 errors 4, no inode ref

root 5 inode 44270613 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1814 namelen 4 name romm filetype 2 errors 4, no inode ref

root 5 inode 44270614 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1815 namelen 8 name docuwiki filetype 2 errors 4, no inode ref

root 5 inode 44270615 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1816 namelen 13 name vscode-server filetype 2 errors 4, no inode ref

root 5 inode 44270616 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1817 namelen 10 name otter-wiki filetype 2 errors 4, no inode ref

root 5 inode 44270617 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1818 namelen 11 name cloudflared filetype 2 errors 4, no inode ref

root 5 inode 44270618 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1819 namelen 19 name postgresql17-picsur filetype 2 errors 4, no inode ref

root 5 inode 44274167 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1820 namelen 9 name mariadb-1 filetype 2 errors 4, no inode ref

root 5 inode 44275288 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1821 namelen 11 name openproject filetype 2 errors 4, no inode ref

root 5 inode 44277617 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1822 namelen 21 name postgresql16-ghosties filetype 2 errors 4, no inode ref

root 5 inode 44278937 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1823 namelen 10 name linkwarden filetype 2 errors 4, no inode ref

root 5 inode 44279172 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1824 namelen 14 name movie-roulette filetype 2 errors 4, no inode ref

root 5 inode 44279184 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1825 namelen 19 name binhex-flaresolverr filetype 2 errors 4, no inode ref

root 5 inode 44279187 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1826 namelen 11 name projectsend filetype 2 errors 4, no inode ref

root 5 inode 44279271 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1827 namelen 10 name dailynotes filetype 2 errors 4, no inode ref

root 5 inode 44279274 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1828 namelen 5 name mysql filetype 2 errors 4, no inode ref

root 5 inode 44279277 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1829 namelen 24 name mariadb-official-project filetype 2 errors 4, no inode ref

root 5 inode 44279278 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1830 namelen 4 name data filetype 2 errors 4, no inode ref

root 5 inode 44279279 errors 2001, no inode item, link count wrong

unresolved ref dir 259 index 1831 namelen 9 name portainer filetype 2 errors 4, no inode ref

ERROR: errors found in fs roots

Opening filesystem to check...

Checking filesystem on /dev/nvme0n1p1

UUID: 1201dc83-793d-417f-a350-31aeb72a9d06

found 575727116288 bytes used, error(s) found

total csum bytes: 455042292

total tree bytes: 823869440

total fs tree bytes: 220495872

total extent tree bytes: 106430464

btree space waste bytes: 102635615

file data blocks allocated: 1015726616576

referenced 529855303680

Post new diags after array start.

friday-diagnostics-20251013-1247.zip

btrfs is detecting data corruption on both pools. Before doing anything else, recommend running memtest

Then, to try and fix that pool, type:

btrfs rescue zero-log /dev/nvme0n1p1

ok i left the mem test running while i was at work and its now been going for 10 hours and 45 minutes still running its on pass 6 29% complete and its status has failed with 40911 errors so i assume there's a memory issue here too. its quite late now ill leave the mem test running over night not sure how long its supposed to take. tomorrow ill find the culprit memory stick and pull that then continue with the btrfs rescue command!

2 hours ago, Uncommon said:

40911 errors

Even 1 error is unacceptable.

You shouldn't even attempt to run any computer unless RAM is working perfectly. Everything goes through RAM. The OS and other executable code, your data. EVERYTHING! The CPU can't do anything with anything until it is loaded into RAM.

ok so after messing with the ram and checking all 4 sticks individually it was a brand new brought 2 months ago stick that was throwing all the errors, i should have tested my ram when i first installed it...

but anyway while swapping ram around i must have knocked one of the sata cables as when i booted it back up one of my 6tb hdds where not detected so i shut to down and checked the sata connector of the hdd is manged no more plastic just the bare metal pins useable and idk if its fixable but thats something for another day. now between the other hdds in the array there more then enough space for all the data that was on that 6tb to be spread across them is there away to start the array without it and have it recover the data on to the other hdds then il replace that drive in a few months or if i need to add a drive to replace it i have a spare 2 tb drive to add in would that work to recover the data and then?
i know there was about 3tb of data on this drive.

what is my best course of action here?

If parity is valid, you can unassign that disk and it will be emulated.

dose the array need to be in maintenance mode?

If the device is mounted, you typically don't need to zero the log, that's only when it can't mount due to a corrupt log.

tried both btrfs rescue /dev/nvme0n1p and `btrfs rescue /dev/nvme1n1` results are in the image what dose this mean i removed the zero log stuff.

i just want to be though as i dont want to mess it up

Post new diags with the array started first.

friday-diagnostics-20251016-0744.zip

Try btrfs rescue zero-log /dev/nvme1n1p1

2 minutes ago, JorgeB said:

Try btrfs rescue zero-log /dev/nvme1n1p1

i also realised you may want fresh digs after this so i reran the commend and get digs after that.

friday-diagnostics-20251016-0907.zip

Restart the array and hopefully that pool will now mount, if it does, you should run a scrub on both pools and delete any corrupt files. Some corrupt files are expected due to the bad RAM, and they will be listed on the syslog after the scrub.