EH5

I forgot to mention I checked the Immich config and it looked fine as far as I could tell. Immich has been working fine for a while.

I removed power from the troublesome drive and rebooted. I got the memory notice again. I hate to be dense but I can't figure out how to control the memory used by Immich. I found the menu to control CPU but not memory. Which menu has that option? I have attached the diagnostics again just in case tower-diagnostics-20240701-1347.zip

My server has 32 Gigs of ram. I was preclearing one disc and running an extended smart test on another when I got the out of memory message. message. The fix common problems tool says to post the diagnostics here to figure out what to do if anything. tower-diagnostics-20240629-1802.zip

I upgrade my power supply to get sufficient sata power connections and switched to a case with easier access to swap out the drives. Tings appeared fine but I got a memory error. I will start a new thread for that as I think all the issues I was dealing with in this tread are fixed.

I moved everything off the emulated disks and unplugged them. When I reconfigured without them and tried to restart the array and rebuild parity it shows that another disk is disconnected, the Dockers didn't start, and the same issue where the CPU Usage diagnostics don't show and the parity rebuild is not advancing. Is it possible that my UNRAID USB drive is corrupted somehow? tower-diagnostics-20240621-1847.zip

Yes there are power splitters. It is a 600W power supply but not enough HDD plugs

I disconnected that drive (WD-WX11D19A4C2V) and the system started. It looks like drive 3 and 4 of the array may just be toast. Should I just remove them from the array? what is the proper procedure for this? tower-diagnostics-20240619-1037.zip

WD-WX11D19A4C2V was one of the non-array disks I was testing to see if it would pass preclear

I shut down from the array operation menu on the Main tab and swapped the cables on disk 3. When I powered up it indicated that it was not a clean shut down. Interestingly, when I checked the notices it reported successful parity check from a couple of days ago. Now disk 3 and 4 show failures and the array just says starting but has not started. the fix common problems just says disk 4 is disabled and refers me to the main tab. tower-diagnostics-20240618-1808.zip

I had been transferring files from portable drives to the array and moving files off of a drive in the that has issues before shutting the array down to fiddle with the failed drive when, after a break a parity check had started. The CPU was at nearly 100% utilization which slowed everything down so I paused the parity check to finish what I was doing and restarted the parity check. It says running but is making no progress and the CPU utilization graphics on the Dashboard don't seem to be working any more. I tried to pause the parity check and even cancel it but after confirming there is no impact on the condition of the parity check. I am including the current diagnostic file. I didn't want to try a reboot until I check here. tower-diagnostics-20240618-1058.zip

That got the dockers up and running. The missing disk is not 366. The 366 disk is the one I installed to run preclear (it failed) so I think it is a bad disk. I will run the extended smart test. The missing disk is actually not showing up anywhere. That is why I was considering shutting down and checking all the connections. I was thinking either something is loose or it really failed hard. Just to confirm I should run check file system a second time on disk 6? In the UI it looks ok right now. Thanks for your help.

Here is the new diagnostic file. I have not rebooted the system since I got disk 6 back up tower-diagnostics-20240604-0741.zip

Disk 6 seems to have been recovered but the Dockers have still not started. Should I just follow the instructions I see on other docker failed to start posts? Also, disk3 lists as not installed. I could shut down the server and try to re-seat the power and data cables. Is that a good idea? I was able to click on the icon next to disk 3 and see all the files so it seems like they are emulated even though there is no notice listing the drive as emulated. Would it be reasonable to use unballanced to move everything off drive3 before doing anything else?

I could not figure out the formatting so here is a text file which may be easier to read Filesystemreport.txt

Here is the top of the report: <?>Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... Metadata CRC error detected at 0x47191d, xfs_inobt block 0x18/0x1000 btree block 0/3 is suspect, error -74 root inode chunk not found Phase 3 - for each AG... - scan and clear agi unlinked lists... found inodes not in the inode allocation tree found inodes not in the inode allocation tree - process known inodes and perform inode discovery... - agno = 0 148c620e4680: Badness in key lookup (length) bp=(bno 0x1b800, len 4096 bytes) key=(bno 0x1b800, len 16384 bytes) 148c620e4680: Badness in key lookup (length) bp=(bno 0x1b820, len 4096 bytes) key=(bno 0x1b820, len 16384 bytes) 148c620e4680: Badness in key lookup (length) bp=(bno 0x491a80, len 4096 bytes) key=(bno 0x491a80, len 16384 bytes) 148c620e4680: Badness in key lookup (length) bp=(bno 0x491aa0, len 4096 bytes) key=(bno 0x491aa0, len 16384 bytes) 148c620e4680: Badness in key lookup (length) bp=(bno 0x8ae040, len 4096 bytes) key=(bno 0x8ae040, len 16384 bytes) 148c620e4680: Badness in key lookup (length) bp=(bno 0x8ae060, len 4096 bytes) key=(bno 0x8ae060, len 16384 bytes) 148c620e4680: Badness in key lookup (length) bp=(bno 0xf1d180, len 4096 bytes) key=(bno 0xf1d180, len 16384 bytes)<?>

Here is the bottom of the resulting report <?>correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 1 - agno = 7 - agno = 6 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done

I removed a disk that failed preclear and added another one to the system to preclear that. when I started up the system one of the array drives was disabled and that data was emulated (i may not have the correct vocabulary for this). I checked the forums and guessed that maybe i bumped the power or data cable for this drive so I reseated everything and started the array again. when it booted it gave me the option of adding that disk in its old slot in the array and rebuilding to it so I did this. After this started I realized the Dockers had not started up. When the rebuild was underway another disk in the array seems to have failed. When the rebuild finished I tried to get the dockers working again by doing a reboot of the system. When the array started this time I got the "unmountable unsupported or no file system" error and the dockers failed to start. Also I do not see any of the warnings about the data being emulated witch is concerning. tower-diagnostics-20240602-1333.zip

I do have the drive. how would I try to access the data on that drive?

I did the pre clear thing before starting the disk. Was this just bad luck or should I have done something different when I set my system up? What can I do to help prevent data loss besides another system to act as a backup?

I reran the system check with the -L and I got some folders including lost+found. There is nothing in the other folder but lost+found has thousands of numbered directories and a few hundred numbered files. The total used space on the drive listed in the main tab is far less than what it should be. What can I do next or am I out of luck on my files that were stored on this drive?

I ran memtest over night and was in the fourth pass with no errors this morning. Ran File System Check Status without the -n and got this result: xfs_repair status: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... Log inconsistent or not a log (last==0, first!=1) empty log check failed zero_log: cannot find log head/tail (xlog_find_tail=22) ERROR: The log head and/or tail cannot be discovered. Attempt to mount the filesystem to replay the log or use the -L option to destroy the log and attempt a repair. What should I try next? Re-run the file system check status with a -L?

Hardware: BIOSTAR Group TP55 American Megatrends Inc., Version 080015 BIOS dated: Mon 14 Dec 2009 12:00:00 AM CST Intel® Core™ i7 CPU 860 @ 2.80GHz Memory: 16 GiB SDRAM Gigabit Dual NIC with Intel 82576 Chip, 1Gb Network Card StorageTekPro Flashed Dell H310 6Gbps SAS HBA to LSI 9211-8i P20 IT Mode for ZFS FreeNAS unRAID Cable Matters Internal Mini SAS to SATA Cable 3.3 Feet 600W Thermaltake Smart Series 80 plus power supply the original "failed" disc was an 8TB WD red I still have the original disk but have not reinstalled it as an individual drive yet There are 6 HDDs of various types in the array including this one and the parity disk. Plugins: Community Applications Fix Common Problems Unassigned Devices Unassigned Devices Plus Unassigned Devices Preclear Unraid Connect UNRAID version: 6.12.4 The Story: I have been working on this project for several months now and have made many mistakes including wrong type of boot drive, USB3 rather than USB2 plug, wrong LAN setup, bad LAN cables, not preclearing drives first (initially), set up shares which could only be accessed one at a time from windows machines, etc. I thought I had sorted things out and got the system stable when the largest array drive seemed to fail. It triggered an error message which I can't remember exactly and was no longer mounted. The drive listed as emulated. I followed some instructions from the internet for replacing a failed array drive and rebuilt to a new 8TB WD RED PRO drive. The rebuild seemed to work fine until it completed and the drive was listed as "Unmountable: Unsupported or no file system". I then did more research and found the links related to file system errors. Based on what I am reading I should have tried more steps before just rebuilding the drive. I had precleared the HDD that failed drive before I had started it up. The results of the file system check are attached along with the system log and the diagnostic zip file. The file system check is scary long so I decided to stop messing around and post here before doing anything else. Some other things I did that may be issues include: I added a new cache drive at the same time I replaced the failed drive (since the case was open anyway). I didn't think this would affect anything but I don't really know. The new cache drive has the same "Unmountable: Unsupported or no file system" which is odd. I also clicked update on the plugins any time a message to do so popped up. Now I am thinking maybe I should have waited until I had stopped the array before clicking those. I saw in another post there was a recommendation to reinstall the "failed" HDD and see if it can be mounted independent of the array. I have the disk available so that is an option. tower-syslog-20231125-2252.zip tower-diagnostics-20231125-1622.zip Disc Check Results.txt