PsyCl0ne

Alright I got it sorted. I dont have proof of exactly what the issue was but by doing the following I was able to get everything working. 1) As per Fix Common Problems, I went into each share, deleted the last number for "Minimum free space" put that number back and applied the change. 2) The drives that I am removing from my system since they are listed under "Unassigned Devices" I had to mount all of them and I deleted the only file that I could see which was "data" after once again checking that there were no files in them. To do this I used File Manager in the unraid GUI. 3) I unmounted the drives and stopped the array. 4) Navigating into Settings > Global Share Settings I was now able to see all the drives I am keeping listed in the drop-down menu for "Included disk(s)". I enabled them all and applied. Then I hit the Done button. 5) Started the array back up and I got one more notification from Fix Common Problems for the ISOS share. I repeated step 1 to correct that issue. 6) Confirmed that all the missing directories were now visible under /mnt/user/. 7) Stopped the array once again, went back into the Global Share Settings, deselected all the drives from the inclusion list so that it showed all. Applied the change and then hit the Done button. 8)) Started the array once more and going into every share settings I can finally see all of my array drives in the Inclusion and Exclusion drop-down menus. 9) Quick skim through my data directory, it appears to now have what I would normally have expected prior to this mess.

Ah kk perfect. Apologies I started to doubt my response. But I'm glad we're getting some traction on this.

Unraid: 6.12.10, setup to be able to complete atomic moves. A couple of nights ago I went through the shrink array guide to remove multiple drives. My array setup at the time was a total of 17 drives. #1-8 were 4TB, #9-11 were 18TB, #12-13 were 4TB, and #14-17 were 18TB. I also have 2 parity drives. The goal was to remove all but 1 of the 4TB drives. Data from each of the to be removed drives were migrated over to a 18TB drive using unbalance, and I confirmed that there was no data left on the drives by browsing to the drive to see if there was anything left over. After completing the steps in "New Config" tool, and moving over to the main tab of unraid, I removed the drives I didnt want and rearranged the drives I wanted to keep starting from Disk 1. Parity rebuild has completed. PROBLEM: Navigating to /mnt/user/data/(Any folder) I see a tiny portion of directories that I would be expecting to see. Navigating to specific disks (/mnt/disk#/data/...) all the directories and all the files that I would expect to see are there. Permissions seem to be correct all the way down from /mnt/disk#/data to whichever directory I navigate to (drwxrwxrwx+ 2 nobody users). Fix Common Problems has a bunch of errors, but I am not sure what to do with them, as the shares appear to be showing correctly instead of what I changed when starting the shrink array guide steps. Although one thing that I do see is that when selecting the inclusion drop down list, there are no drives listed to select. Screenshots of Fix Common Problems and each of the share settings are attached. Last screenshot is of the global share settings. What is it that I have missed, making this process not complete properly? tower-diagnostics-20240412-1441.zip

@JorgeB Thank you, honestly you're a godsend. Thank you for your help once again.

JorgeB, apologies your help has been appreciated but while I understand what you have advised I am not sure of the steps to take. I know that normally with a failed drive if I take it out and then put in a new drive, I can rebuild the array. How would it work in this situation? Do I force the system to start a rebuild or do I take the drive out and start the rebuild? Would the array rebuild onto the 4 TB drives that are still in shape? Attached are screenshots of how my "Main" tab is looking for the drives and array options. A side note when I clicked on the little icon next to the drive it pulled up Logs? and I have the following showing in it for the drive that is disabled Nov 1 05:30:04 Tower emhttpd: spinning down /dev/sdo Nov 1 06:00:01 Tower emhttpd: read SMART /dev/sdo Nov 1 07:30:04 Tower emhttpd: spinning down /dev/sdo Nov 1 08:00:01 Tower emhttpd: read SMART /dev/sdo Nov 1 08:30:23 Tower emhttpd: spinning down /dev/sdo Nov 1 09:00:01 Tower emhttpd: read SMART /dev/sdo Nov 1 10:30:03 Tower emhttpd: spinning down /dev/sdo Nov 1 11:00:16 Tower kernel: sd 8:0:1:0: [sdo] tag#2775 CDB: opcode=0x85 85 06 20 00 00 00 00 00 00 00 00 00 00 40 e5 00 Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2798 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=19s Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2798 Sense Key : 0x2 [current] Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2798 ASC=0x4 ASCQ=0x0 Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2798 CDB: opcode=0x88 88 00 00 00 00 06 33 f5 63 d0 00 00 04 00 00 00 Nov 1 11:00:20 Tower kernel: I/O error, dev sdo, sector 26641523664 op 0x0:(READ) flags 0x0 phys_seg 128 prio class 0 Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2799 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=19s Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2799 Sense Key : 0x2 [current] Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2799 ASC=0x4 ASCQ=0x0 Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2799 CDB: opcode=0x88 88 00 00 00 00 06 33 f5 5b d0 00 00 04 00 00 00 Nov 1 11:00:20 Tower kernel: I/O error, dev sdo, sector 26641521616 op 0x0:(READ) flags 0x0 phys_seg 128 prio class 0 Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2800 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=19s Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2800 Sense Key : 0x2 [current] Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2800 ASC=0x4 ASCQ=0x0 Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2800 CDB: opcode=0x88 88 00 00 00 00 06 33 f5 5f d0 00 00 04 00 00 00 Nov 1 11:00:20 Tower kernel: I/O error, dev sdo, sector 26641522640 op 0x0:(READ) flags 0x0 phys_seg 128 prio class 0 Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2801 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=19s Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2801 Sense Key : 0x2 [current] Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2801 ASC=0x4 ASCQ=0x0 Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2801 CDB: opcode=0x88 88 00 00 00 00 06 33 f5 4f d0 00 00 04 00 00 00 Nov 1 11:00:20 Tower kernel: I/O error, dev sdo, sector 26641518544 op 0x0:(READ) flags 0x0 phys_seg 128 prio class 0 Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2802 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=19s Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2802 Sense Key : 0x2 [current] Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2802 ASC=0x4 ASCQ=0x0 Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2802 CDB: opcode=0x88 88 00 00 00 00 06 33 f5 53 d0 00 00 04 00 00 00 Nov 1 11:00:20 Tower kernel: I/O error, dev sdo, sector 26641519568 op 0x0:(READ) flags 0x0 phys_seg 128 prio class 0 Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2803 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=19s Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2803 Sense Key : 0x2 [current] Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2803 ASC=0x4 ASCQ=0x0 Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2803 CDB: opcode=0x88 88 00 00 00 00 06 33 f5 57 d0 00 00 04 00 00 00 Nov 1 11:00:20 Tower kernel: I/O error, dev sdo, sector 26641520592 op 0x0:(READ) flags 0x0 phys_seg 128 prio class 0 Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2804 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2804 Sense Key : 0x2 [current] Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2804 ASC=0x4 ASCQ=0x0 Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2804 CDB: opcode=0x88 88 00 00 00 00 06 33 f5 67 d0 00 00 04 00 00 00 Nov 1 11:00:20 Tower kernel: I/O error, dev sdo, sector 26641524688 op 0x0:(READ) flags 0x4000 phys_seg 128 prio class 0 Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2815 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2815 Sense Key : 0x2 [current] Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2815 ASC=0x4 ASCQ=0x0 Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2815 CDB: opcode=0x88 88 00 00 00 00 06 33 f5 6b d0 00 00 04 00 00 00 Nov 1 11:00:20 Tower kernel: I/O error, dev sdo, sector 26641525712 op 0x0:(READ) flags 0x0 phys_seg 128 prio class 0 Nov 1 11:00:20 Tower emhttpd: read SMART /dev/sdo Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2761 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2761 Sense Key : 0x2 [current] Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2761 ASC=0x4 ASCQ=0x0 Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2761 CDB: opcode=0x8a 8a 00 00 00 00 06 33 f5 63 d0 00 00 04 00 00 00 Nov 1 11:00:20 Tower kernel: I/O error, dev sdo, sector 26641523664 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0 Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2765 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2765 Sense Key : 0x2 [current] Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2765 ASC=0x4 ASCQ=0x0 Nov 1 11:00:20 Tower kernel: sd 8:0:1:0: [sdo] tag#2765 CDB: opcode=0x8a 8a 00 00 00 00 06 33 f5 67 d0 00 00 04 00 00 00 Nov 1 11:00:20 Tower kernel: I/O error, dev sdo, sector 26641524688 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0 Nov 1 11:30:23 Tower emhttpd: spinning down /dev/sdo Nov 2 10:24:06 Tower emhttpd: read SMART /dev/sdo Nov 2 10:27:07 Tower emhttpd: shcmd (6623718): echo 128 > /sys/block/sdo/queue/nr_requests

Hey thank you for looking into the logs. What is the next best step? Reboot the server? Still swap the drive out? Am I able to still pull the data from this drive?

Hey everyone, just got 3 18TB HDDs, Completed preclear, without issue. While migrating from my smaller 4TB drives onto the new drives using unbalance one of the drives disabled itself. I have completed a long SMART test that came back clean so I am not sure what is going on. Attached are logs from the diagnostics page as well as the SMART log from the drive itself. Second question to go along with this, if the drive has actually shit the bed, can I use unbalance again to move the data off the drive back onto the 4TB drives or what is the best way to go about this? I have a new 18TB drive on its way but it will be another ~4 days at least. tower-smart-20231103-1459.zip tower-diagnostics-20231103-1456.zip

Alright so quick update, I have swapped the CPUs in their respective sockets after cleaning them up. Cleared the IPMI logs and have just got Unraid back up and running. Lets see what comes up in the logs next, but from the basic googling I have done it seems more likely that the PSU is starting to fail rather than the CPUs. On that topic, does anyone know where I could get a replacement Segate ss-400h2u? From what I am seeing in CAD its like $400-$500 "new". Or does anyone know if a Segate ss-600h2u would be swapable? I know that desktop PSUs are not to be trusted due to the cabling variations, regardless of brand or model. Does the same stand for server PSUs?

Apologies for the delayed response. I noticed that while in the IPMI I was unable to see any sensor information at all. Everything was showing and unknown/unavailable. What I have done since is actually reflash the BIOS and Firmware for my mobo, and now I am seeing proper errors and data from the sensors. I have attached all that I can from the system information screen and a new diagnostics output from unraid. From what I can see, IPMI log seems to be filling up with a bunch of CPU lines, but I am not 100% just yet on what it means. I wanted to get the info posted here before trying to figure out what it's trying to explain. Thank you for the reply, I'm not sure if I would have much faith in my equipment as I am still learning quite a bit and looks like there is still a ton more to learn. So maybe take my post with a grain of salt, could currently just be happenstance. IPMI Event Log.xlsx IPMI Sensor Readings + Threshholds.xlsx tower-diagnostics-20230312-1635.zip

Apologies, I don't follow. Its a dual socket motherboard, and both sockets have a X5670 CPU in them. Are you telling me to swap the CPUs in each socket? This server has been running for at least a couple of years in this config, if that is of any help.

Perfect, I figured out how to get the IPMI going. The only thing as of this year was Event ID Time Stamp Sensor Name Sensor Type Description 251 12/04/2023 16:41:04 Unknown [undefined] undefined - Asserted 250 09/18/2021 15:05:04 Unknown OS Critical Stop undefined - Asserted - Asserted

Apologies, how would I locate these logs? Is it the ones located at http://tower/log/syslog? If so, I have it attached, however I don't think I am seeing anything that stands out. It seems to be populated from after the server reset. syslog

Hey everyone, My server just crashed while I was streaming on Jellyfin, however this is a really strange crash as my server didn't fully restart. I'm not sure how to explain this, but I didn't hear it go through post and ramping up the chassis fans and then back down. I just heard the fans spin up for a moment and then go back down. However, unraid did have to boot back up, and I was able to connect to it once it was showing the IP address. The server is currently completing a parity check but in fix common problems I have the following error "Machine Check Events detected on your server" "Your server has detected hardware errors. You should install mcelog via the NerdPack plugin, post your diagnostics and ask for assistance on the Unraid forums. The output of mcelog (if installed) has been logged More Information" Trying to get more information for this error but installing NerdPack, I am unable to find that in the app store, I believe it was depreciated? There is, however, NerdTools, but I am unable to find that mcelog tool within it. I have attached the diagnostics from when the UI was accessible, but I am not sure if that would have information as to what happened. Any idea what direction I should move with this? tower-diagnostics-20230306-2238.zip

Thank you both, I don't have much experience with server hardware and didn't know that there was an option within the HBA to enable drives to boot from but it is sorted now! If anyone else does come across this issue, I have the following hardware: SAS9211-8i + SAS9207-8e. Info for the boot settings was found from https://support.osnexus.com/hc/en-us/articles/212131146-Setting-LSI-SAS-HBA-boot-support

In the bios it is seeing all of my drives and the USB drive but it just won't boot off of it...

There is only one device enabled to boot off of.

Fair point, I have the server set to only boot off of the USB stick and nothing else. Even rebooting the server and getting into the boot menu and selecting the USB drive ends up with the same result.

Hey everyone, Recently I purchased some WD Reds 4 TB WD40EFZX drives to fill up the last slot I have on my server but when I have the drive inserted into the hot swap bay the server displays an error "Reboot and Select proper boot device or Insert Boot Media in selected Boot device and press a key" When I remove the new HDD the server boots fine. When Unraid is fully booted, and I insert the drive, it doesn't get detected, at least from what I can tell. I have tried the two internal USB ports as well as the two USB ports on the rear IO panel of the motherboard. I have no idea if this is an issue with the new drives, or maybe a configuration setting that I just don't know about. Currently, the server boots with 11 drives ( 2 SSD cache, 9 4TB mix between Seagate, HGST, WD). For more in formation I have attached the diag zip but if there is anything else I can provide please let me know! tower-diagnostics-20220227-2303.zip

Thank you for your time JorgeB. Attached the new diagnostics after following your steps. I did not reboot just yet but will shortly after posting this. tower-diagnostics-20210820-2115.zip

Randomly I have lost the ability to start any dockers besides netdata. No updates or randomly power outages. Everything was working fine server uptime of 31 days. When trying to refresh web interfaces for dockers that were running, I was seeing some SQL errors. I tried to restart the docker service and now nothing will start up. Last time I was having issues I was advised to delete and recreate the docker image which worked so its a fairly new image. I don't think I have any hardware failure happening, as fix common problems doesn't see anything. Any advice would be appreciated. Diagnostics have been attached. tower-diagnostics-20210819-2149.zip

After a recent random cut of power to my server I have been experiencing a bunch of issues. After the power outage the server came back and completed a check no problem. Followed by this I then saw both my docker image and logs being filled to 100%. Docker was running without issues prior to this. I tried to reboot the server and the first reboot everything came back clean and after starting the array everything fired up but shortly after the docker image was full again. Logs were fine at this point and sitting at ~1% as per gui. So I tried to reboot the server again just to see if I could get into some of the recently updated dockers to see if I had a config issue, but now the docker service won't even start. One thing that I did notice is that the reboot took forever to happen. I am not sure where to even begin with diagnostics here, so any help would be appreciated. Attached are my diagnostics logs. tower-diagnostics-20210715-0012.zip

Thank you very much for your help! I tried rebuilding to the same drive and noticed that the reallocated sectors grew to 2136. I have stopped the rebuild and have replaced the drive. Just need to preclear and then I should be able to start the rebuild. I just wanted to thank you both once again! Please take care and see you around!!

Hey johnnie.black, Thank you for your reply. I honestly appreciate it. Quick question if you dont mind. How do I get my array back up and running currently its appearing as faulty and the drives contents are being emulated. I took a look at the settings of the drive itself and I dont see anything that sticks out. Ive attached a small screenshot of what the error is.

Hey everyone, I have a drive that recently started acting up. Temp warning were appearing up to I think it was 123 C even though its sitting in a 4u server with proper cooling. Even ambient temp in the room was 19C. Anyway now its been marked as faulty but running the extended SMART test came back fine. Can I please get some assistance figuring out what is going on? Please and thank you in advance! tower-diagnostics-20180929-0044.zip tower-smart-20180927-0304.zip tower-smart-20180928-1555.zip