DivideBy0

Jorge I appreciate the followup and that is great news the xfs_repair will address the issue. A lot of moving parts for sure, hard work and data is very important to all of us here. Thanks again Sir

LOL the question was not about me but thanks for taking the time stereotyping me I am a zero tolerance kind of guy and given the data I have I won't tolerate any mistakes, if I can avoid them, so my question was pretty fair I think, half or not of a glass The answer should be, nothing is safe, lots of moving parts, here is how you can clean you filesystem, just in case, make sure you have backups, and if shite happens we will investigate and improve and let you know

Now that's troubling Should we all expect this when upgrading from 6.9.2 to 6.11.0? And is there a way to clean / clear the corruption before upgrading? What will the path forward be?

Will do. I may just write a quick script to dump the rootfs df to the remote syslog file. I am sure I will catch something. This is the second time it crashed like this. Thanks again Sir

root@NAS-UNRAID-BKP:/# ls -lah /mnt total 16K drwxr-xr-x 14 root root 280 Sep 10 10:36 ./ drwxr-xr-x 21 root root 460 Sep 10 12:51 ../ drwxrwxrwx 1 nobody users 52 Sep 10 10:46 cache/ drwxrwxrwx 3 nobody users 20 Sep 10 10:46 disk1/ drwxrwxrwx 9 nobody users 131 Sep 10 10:46 disk2/ drwxrwxrwx 3 nobody users 20 Sep 10 10:46 disk3/ drwxrwxrwx 7 nobody users 98 Sep 10 10:46 disk5/ drwxrwxrwx 3 nobody users 21 Sep 10 10:46 disk6/ drwxrwxrwx 4 nobody users 50 Sep 10 10:46 disk7/ drwxrwxrwt 2 nobody users 40 Sep 10 10:36 disks/ drwxrwxrwt 2 nobody users 40 Sep 10 10:36 remotes/ drwxrwxrwt 2 nobody users 40 Sep 10 10:36 rootshare/ drwxrwxrwx 1 nobody users 20 Sep 10 10:46 user/ drwxrwxrwx 1 nobody users 20 Sep 10 10:46 user0/ root@NAS-UNRAID-BKP:/#

I have the same thing as you: bin/ boot/ dev/ etc/ home/ hugetlbfs/ init@ lib/ lib64/ mnt/ opt/ proc/ root/ run/ sbin/ sys/ tmp/ usr/ var/

I have the scripts plugin installed but don't run or schedule any scripts at all

All dockers are stopped now during parity check. But only pi-hole, net data and transmission are running all the time, all others are stoped

Thanks much Sir That's weird. I wasn't doing anything different or weird other than just normal running? It was up for like 90 days? Is there a way to keep an eye on rootfs? I am kind of annoyed with this repeating issue My primary unRaid don't have this same issue. Wonder what is different as they're identically the same setup same plugins root@NAS-UNRAID-BKP:~# df -h / Filesystem Size Used Avail Use% Mounted on rootfs 32G 1.1G 31G 4% /

My GUI was crashed with some weird php dynamix error. I did a reboot at command line since GUI didn't work, it came back clean, but it started a parity check. I think this is the second time I have this problem. See attached diagnostic file created at reboot please. Any ideas? I think this is the problem here? Sep 10 08:49:58 NAS-UNRAID-BKP emhttpd: spinning down /dev/sdj Sep 10 08:49:58 NAS-UNRAID-BKP emhttpd: spinning down /dev/sdk Sep 10 08:49:58 NAS-UNRAID-BKP emhttpd: spinning down /dev/sdh Sep 10 10:12:30 NAS-UNRAID-BKP emhttpd: read SMART /dev/sdj Sep 10 10:12:30 NAS-UNRAID-BKP emhttpd: read SMART /dev/sdh Sep 10 10:12:30 NAS-UNRAID-BKP emhttpd: read SMART /dev/sde Sep 10 10:28:44 NAS-UNRAID-BKP root: error: /plugins/community.applications/scripts/notices.php: uninitialized csrf_token Sep 10 10:28:44 NAS-UNRAID-BKP root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token ### [PREVIOUS LINE REPEATED 3 TIMES] ### Sep 10 10:28:44 NAS-UNRAID-BKP root: error: /webGui/include/DashboardApps.php: uninitialized csrf_token Sep 10 10:28:44 NAS-UNRAID-BKP root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token ### [PREVIOUS LINE REPEATED 1 TIMES] ### Sep 10 10:28:44 NAS-UNRAID-BKP root: error: /webGui/include/Notify.php: uninitialized csrf_token Sep 10 10:28:44 NAS-UNRAID-BKP root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token ### [PREVIOUS LINE REPEATED 1 TIMES] ### Sep 10 10:28:44 NAS-UNRAID-BKP root: error: /webGui/include/Notify.php: uninitialized csrf_token Sep 10 10:28:49 NAS-UNRAID-BKP root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token ### [PREVIOUS LINE REPEATED 4 TIMES] ### Sep 10 10:29:00 NAS-UNRAID-BKP unassigned.devices: PHP Warning: parse_ini_file(/usr/local/emhttp/state/users.ini): failed to open stream: No such file or directory in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 48 Sep 10 10:29:00 NAS-UNRAID-BKP unassigned.devices: PHP Warning: parse_ini_file(/usr/local/emhttp/state/disks.ini): failed to open stream: No such file or directory in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 49 Sep 10 10:29:00 NAS-UNRAID-BKP unassigned.devices: PHP Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/dynamix/include/DefaultPageLayout.php(522) : eval()'d code on line 19 Sep 10 10:29:00 NAS-UNRAID-BKP root: error: /webGui/include/DeviceList.php: uninitialized csrf_token ### [PREVIOUS LINE REPEATED 2 TIMES] ### Sep 10 10:29:00 NAS-UNRAID-BKP root: error: /plugins/community.applications/scripts/notices.php: uninitialized csrf_token Sep 10 10:29:00 NAS-UNRAID-BKP root: error: /plugins/dynamix.plugin.manager/scripts/PluginAPI.php: uninitialized csrf_token Sep 10 10:29:00 NAS-UNRAID-BKP root: error: /plugins/unassigned.devices/UnassignedDevices.php: uninitialized csrf_token Sep 10 10:29:00 NAS-UNRAID-BKP root: error: /webGui/include/Notify.php: uninitialized csrf_token Sep 10 10:29:00 NAS-UNRAID-BKP root: error: /plugins/unassigned.devices/UnassignedDevices.php: uninitialized csrf_token Sep 10 10:29:00 NAS-UNRAID-BKP root: error: /webGui/include/Notify.php: uninitialized csrf_token Sep 10 10:29:03 NAS-UNRAID-BKP root: error: /webGui/include/DeviceList.php: uninitialized csrf_token Sep 10 10:29:03 NAS-UNRAID-BKP root: error: /plugins/unassigned.devices/UnassignedDevices.php: uninitialized csrf_token Sep 10 10:29:06 NAS-UNRAID-BKP root: error: /webGui/include/DeviceList.php: uninitialized csrf_token Sep 10 10:29:06 NAS-UNRAID-BKP root: error: /plugins/unassigned.devices/UnassignedDevices.php: uninitialized csrf_token ### [PREVIOUS LINE REPEATED 1 TIMES] ### Sep 10 10:29:09 NAS-UNRAID-BKP root: error: /webGui/include/DeviceList.php: uninitialized csrf_token Sep 10 10:29:11 NAS-UNRAID-BKP root: error: /plugins/community.applications/scripts/notices.php: uninitialized csrf_token Sep 10 10:29:11 NAS-UNRAID-BKP root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token ### [PREVIOUS LINE REPEATED 5 TIMES] ### Sep 10 10:29:11 NAS-UNRAID-BKP root: error: /webGui/include/DashboardApps.php: uninitialized csrf_token Sep 10 10:29:11 NAS-UNRAID-BKP root: error: /webGui/include/Notify.php: uninitialized csrf_token Sep 10 10:29:11 NAS-UNRAID-BKP root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token ### [PREVIOUS LINE REPEATED 1 TIMES] ### Sep 10 10:29:11 NAS-UNRAID-BKP root: error: /webGui/include/Notify.php: uninitialized csrf_token Sep 10 10:29:16 NAS-UNRAID-BKP root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Sep 10 10:29:19 NAS-UNRAID-BKP emhttpd: spinning down /dev/sdj Sep 10 10:29:19 NAS-UNRAID-BKP emhttpd: spinning down /dev/sdh Sep 10 10:29:19 NAS-UNRAID-BKP emhttpd: spinning down /dev/sde Sep 10 10:29:21 NAS-UNRAID-BKP root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Sep 10 10:29:22 NAS-UNRAID-BKP sshd[17416]: Connection from 192.168.0.7 port 50098 on 192.168.0.13 port 22 rdomain "" Sep 10 10:29:22 NAS-UNRAID-BKP sshd[17416]: Postponed keyboard-interactive for root from 192.168.0.7 port 50098 ssh2 [preauth] Sep 10 10:29:24 NAS-UNRAID-BKP sshd[17416]: Postponed keyboard-interactive/pam for root from 192.168.0.7 port 50098 ssh2 [preauth] Sep 10 10:29:24 NAS-UNRAID-BKP sshd[17416]: Accepted keyboard-interactive/pam for root from 192.168.0.7 port 50098 ssh2 Sep 10 10:29:24 NAS-UNRAID-BKP sshd[17416]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0) Sep 10 10:29:24 NAS-UNRAID-BKP sshd[17416]: Starting session: shell on pts/1 for root from 192.168.0.7 port 50098 id 0 Sep 10 10:29:26 NAS-UNRAID-BKP root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token ### [PREVIOUS LINE REPEATED 27 TIMES] ### Sep 10 10:30:24 NAS-UNRAID-BKP root: error: /webGui/include/Boot.php: uninitialized csrf_token Sep 10 10:30:40 NAS-UNRAID-BKP login: pam_unix(login:session): session opened for user root(uid=0) by LOGIN(uid=0) Sep 10 10:30:40 NAS-UNRAID-BKP login: ROOT LOGIN ON tty1 Sep 10 10:30:56 NAS-UNRAID-BKP shutdown[20935]: shutting down for system reboot Sep 10 10:30:56 NAS-UNRAID-BKP init: Switching to runlevel: 6 nas-unraid-bkp-diagnostics-20220910-1032.zip

This whole thing is very CONFUSING. Can someone actually show a good working example of the 2 settings? I had a similar situation when transferring 1TB worth of data and my cache got full, system crashed and took a bit to recover from that mistake. I am still not sure how set this honestly.

Agreed, makes sense and is a good reminder. I do have 2 copies offline (hard drive and second UnRaid server) and another copy on cloud. I hope I am covered

Yes I do have backups offsite and cloud for everything important and irreplaceable. But why the question? Do you consider this system unstable and prone to further corruption? The parity check came back with zero errors and now the disconnected drive is being rebuilt as we speak. With that I think the system is stable? Should I attach another diagnostic after all is done?

Just do another USB with memetest on it and boot your box like that. Replace the UnRaid USB of course

Understand. But what path should I take just preventively? Just to be sure. I don't mind blowing ups my cache and reformatting

Makes sense. Currently I have appdata, domains and system in cache for performance reasons. I can put them back on the array. Should I reformat the cache or repair it? In case something got corrupted I prefer to fix it now before data start moving through the cache. What do you advise? root@NAS-UNRAID:/mnt/cache# ls -al total 16 drwxrwxrwx 1 nobody users 40 Jul 19 12:39 ./ drwxr-xr-x 14 root root 280 Jul 19 12:29 ../ drwxrwxrwx 1 nobody users 162 Jul 9 19:24 appdata/ drwxrwxrwx 1 nobody users 0 May 26 2020 domains/ drwxrwxrwx 1 nobody users 26 May 26 2020 system/

Let me see if I got this right. On my share before transferring the 1TB my settings was Yes : Cache for that new share. But when it hit the max of 500GB of my cache everything crashed and my GUI blew off. For sure I've seen some errors in my IPMI console about the cache full or something like that, so it did not spill over to the array. Is that because I had nothing set under the Main -> Pool Device -> Cache -> Minimum free space: 0 ? I have a backup UnRaid and will do a test, because this entire experience hosed my system and ain't good I really appreciate the support and guidance

Well when I go to Main and under cache settings and I look at the help menu I get this which is conflict with your statement? Minimum free space: "This defines a "floor" for the amount of free space remaining in the volume. If the free space becomes less than this value, then new files written via user shares will fail with "not enough space" error." No mention of bypass cache and write directly to array? Am I looking at the right settings? Am

Thank you Sir. I looked at all my docker containers and they all have paths to a /mnt storage point so I should be covered. Now on the cache issue, how does it work? My share was set to use cache for new files, but the size of the transferred files (1TB) was bigger than the cache (500GB). So when it hits 500GB what does it happen. That's where my crash happened. Is either the way the cache behaved or it may be something with the "DirSyncPro" app I used?

Thanks much. I will clean the shares configs after this. For sure it was filling rootfs as I seen some rootfs errors on my IPMI console, so that's it. So after the parity is done I just rebuild disk 2? - Stop the array - Set the disk2 to be not installed - Start the array - Stop the array - Set the disk2 to be the appropriate disk - Start the array - unRaid will now commence the rebuild operation.

Yes that was the df -h after the reboot Ok here is the history to what happened and I am sure it was related. - I created a new share to prep a 1TB transfer from an external SSD drive. I said "Yes to Use cache pool (for new files/directories)" - Plugged in the SSD drive and start the transfer for the 1TB - It looks like my cache drive is only 500GB so at some point during the transfer something happened, cache drive got full, and I started having these dynamix errors and GUI crashed. - I did a command line reboot, it came back and started parity check. During parity check the move script was trying to move from cache to the new share. It did finish moving whatever was on cache, about 480GB. Now cache drive is empty. - I got the dynamix errors again during the parity check, GUI was messed up, I managed to stop and reboot. When back online, disk 2 was disabled. - I uninstalled all my dynamix plugins for now, so my GUI stop crashing. - Now I started a parity check again. I hope after parity check I can rebuild this disk? Data is all there I guess emulated? What course should I take? Wait for parity check to finish, hopefully no errors and then do a disk rebuild? Is it possible my parity got messed up during the unsuccessful transfer from the 1TB SSD drive? Attached diagnostic. nas-unraid-diagnostics-20220717-1937.zip

Well I managed to stop the parity and do a reboot but one of the disks now show disabled? Should I start my parity check again? and hopefully that will fix the disabled disk? root@NAS-UNRAID:~# df -h Filesystem Size Used Avail Use% Mounted on rootfs 16G 755M 15G 5% / devtmpfs 16G 0 16G 0% /dev tmpfs 16G 0 16G 0% /dev/shm cgroup_root 8.0M 0 8.0M 0% /sys/fs/cgroup tmpfs 128M 280K 128M 1% /var/log /dev/sda1 59G 526M 59G 1% /boot overlay 16G 755M 15G 5% /lib/modules overlay 16G 755M 15G 5% /lib/firmware tmpfs 1.0M 0 1.0M 0% /mnt/disks tmpfs 1.0M 0 1.0M 0% /mnt/remotes tmpfs 1.0M 0 1.0M 0% /mnt/rootshare /dev/md1 7.3T 4.4T 3.0T 60% /mnt/disk1 /dev/md2 7.3T 3.2T 4.2T 43% /mnt/disk2 /dev/md3 7.3T 2.6T 4.8T 36% /mnt/disk3 /dev/md4 7.3T 52G 7.3T 1% /mnt/disk4 /dev/md5 7.3T 52G 7.3T 1% /mnt/disk5 /dev/md6 7.3T 52G 7.3T 1% /mnt/disk6 /dev/sdb1 448G 25G 421G 6% /mnt/cache shfs 44T 11T 34T 24% /mnt/user0 shfs 44T 11T 34T 24% /mnt/user /dev/loop2 20G 2.8G 17G 14% /var/lib/docker /dev/loop3 1.0G 3.8M 905M 1% /etc/libvirt

Talking about timing. Right in middle of a parity check and GUI decided to go south on me The good news parity is still running in the background. Any way that I can restart the GUI or just wait for the parity to be over and reboot command line? I am on 6.9.2 by the way And what could cause this GUI crash? see attached pic And get this in syslog: root@NAS-UNRAID:/var/log# tail -f syslog Jul 17 18:51:38 NAS-UNRAID root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Jul 17 18:51:38 NAS-UNRAID root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Jul 17 18:51:41 NAS-UNRAID root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Jul 17 18:51:46 NAS-UNRAID root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Jul 17 18:51:46 NAS-UNRAID root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Jul 17 18:51:46 NAS-UNRAID root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Jul 17 18:51:51 NAS-UNRAID root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Jul 17 18:51:53 NAS-UNRAID root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Jul 17 18:51:53 NAS-UNRAID root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Jul 17 18:51:56 NAS-UNRAID root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token

Maybe they can put some kind of switch in the preferences? Shouldn't be that hard of a fix?

Is it possible to get back my GUI layout in the right up corner as I have in my current 6.9.2 install? I really like the server details I get in 6.9.2 in right upper corner