October 30, 20178 yr I got home from work and noticed my Plex Server wasn't responding, and not my SMB share, and not the WebGUI either. I held the power button down on the unRAID server to shut it down and when it came backup I had "device disabled, contents emulated" on two of disks (three at first, one of which was a parity drive, but that fixed itself after another reboot). All of these disks were connected to my PCI SATA controller, so I'm fairly sure that thing got a spasm, or the SATA cables connecting them got (very old cheap cables, I'm ordering some new ones, I promise) so I think the disks are alright. I also ran a SMART test on them and both were fine, apparently. So I followed the instructions here https://wiki.lime-technology.com/Troubleshooting#Re-enable_the_drive for re-enabling drive(s) and yep, it's doing that now. But I am confused. Do I need to format them as the "unmountable disks present" warning says? It seems to be doing it's job (note the write speeds) so I am very confused.
October 30, 20178 yr Don't format anything!! Please go to Tools -> Diagnostics and post the file so folks can take a more detailed look and offer help.
October 30, 20178 yr Author Here are the diagnostics tower-diagnostics-20171030-2056.zip I think the formatting isn't needed because when it's reconstructing its rebuilding the disk on a hardware/sector by sector basis, so when it's done reconstructing the file system will be back, right? Same reason that my dockers are gone right? They were on Disk 1, but since Disk 1 is disabled they can't be "found", but they'll be back once the disk has been reconstructed? I'm not gonna be home at all tomorrow so I'm hoping this will just sort it self out, I can't really do anything about it right now until the weekend. Edited October 30, 20178 yr by nadbmal
October 30, 20178 yr Community Expert Probably will have to fix filesystem AFTER the rebuild. Definitely DO NOT format anything.
October 31, 20178 yr Author The reconstruction finished without errors, but now it looks like this. What do I do?
October 31, 20178 yr Stop the array and the start the array in maintenance mode. Click on the disk you want to repair, e.g. disk 1 or disk 4. Scroll to the section CHECK FILESYSTEM STATUS and click the Check button. By default it will to a read only scan. See the Help text to do a repair action (in your case remove the -n option).
October 31, 20178 yr Community Expert 19 hours ago, nadbmal said: I think the formatting isn't needed because when it's reconstructing its rebuilding the disk on a hardware/sector by sector basis, so when it's done reconstructing the file system will be back, right? Same reason that my dockers are gone right? They were on Disk 1, but since Disk 1 is disabled they can't be "found", but they'll be back once the disk has been reconstructed? Since I didn't directly answer these questions, I thought it might be good to do so just for edification. Formatting isn't needed, and in fact, must be avoided. Formatting a disk writes an empty filesystem to the disk. This write updates parity just like any other write. So if you format a disk in the parity array, parity will agree that it has an empty filesytem. Then the rebuild will result in an empty filesystem. The filesystem most likely will not be back after the rebuild, since the contents of the disk is already being emulated by parity, and that emulated data will be the result of the rebuild. And it is the fact that the emulated disk is unmountable that prevents it from being read, not the fact that it is disabled.
October 31, 20178 yr Community Expert 4 minutes ago, trurl said: Since I didn't directly answer these questions, I thought it might be good to do so just for edification. Formatting isn't needed, and in fact, must be avoided. Formatting a disk writes an empty filesystem to the disk. This write updates parity just like any other write. So if you format a disk in the parity array, parity will agree that it has an empty filesytem. Then the rebuild will result in an empty filesystem. The filesystem most likely will not be back after the rebuild, since the contents of the disk is already being emulated by parity, and that emulated data will be the result of the rebuild. And it is the fact that the emulated disk is unmountable that prevents it from being read, not the fact that it is disabled. I'm amazed at how may users think formatting will do anything other than just delete all data on that disk, I'm glad LT added to the warning starting on v6.4rc10, though I'm sure some will still do it anyway.
October 31, 20178 yr 3 minutes ago, johnnie.black said: I'm glad LT added to the warning starting on v6.4rc10, though I'm sure some will still do it anyway. In the next version the warning will show the moment you hover over the checkbox, making it even more prominent. Time will tell how foolproof this is!
October 31, 20178 yr Author Doing the xfs_repair on both Disk 1 and 4 now, we'll see how it goes. All it immediately said was Phase 1 - find and verify superblock... bad primary superblock - bad CRC in superblock !!! but I assume that's expected. I'll let you know how it goes. In the meantime, I'm trying to figure out what happened that caused this to happen in the first place. I'm 99% sure it was the PCI SATA card, a Delock PCIe SATA x4, as the disks that failed were the ones that were connected to it. Also note that Parity 2 failed, but came back after a reboot just fine while the two data disks didn't, so I have no idea what's going on there. I did observe that my server had just hit >30 days uptime the day this happened. I remember, because I was gonna make a tweet saying how stable everything had been so far So I'm thinking, maybe this PCI card has problems with longer uptimes? But I don't really see why that would be. But if it was, maybe I could prevent it in the future by scheduling a weekly reboot or something? My other alternative is to find a ATX motherboard with >= 7 SATA ports, and move my unRAID server into a ATX case, but I'm not sure if those motherboards even exist, atleast at a non-enterprise level. This is the part that bothers me the most, because I have no idea what failed. Everything had just hung up. And I would hate for this to be a bi-monthly occurence. Oh and I'm gonna go order some new SATA cables right now, with clips.
October 31, 20178 yr Community Expert 21 minutes ago, nadbmal said: but I assume that's expected. I'll let you know how it goes. Not really, and not a good sign, but maybe it will find a good backup superblock. 21 minutes ago, nadbmal said: In the meantime, I'm trying to figure out what happened that caused this to happen in the first place. I'm 99% sure it was the PCI SATA card, a Delock PCIe SATA x4, as the disks that failed were the ones that were connected to it. That's a Marvell controller and definitely not recommended, as you noticed both disabled disks plus parity2 are connected there, you should replace it with a LSI HBA.
November 1, 20178 yr Author So both of the disks got this now, with the xfs_repair: Phase 1 - find and verify superblock... bad primary superblock - bad CRC in superblock !!! attempting to find secondary superblock... ...(a ton of dots).......................................found candidate secondary superblock... verified secondary superblock... writing modified primary superblock sb realtime bitmap inode 18446744073709551615 (NULLFSINO) inconsistent with calculated value 97 resetting superblock realtime bitmap ino pointer to 97 sb realtime summary inode 18446744073709551615 (NULLFSINO) inconsistent with calculated value 98 resetting superblock realtime summary ino pointer to 98 Phase 2 - using internal log - zero log... ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this. EDIT: So I ran with the -L command, after Googling, and hey, everything is back now... except my Docker containers, all of them are gone. On Disk 1, there is a docker.img file that was modified around the time I got home from work and noticed everything had hung up. Any ideas? I would really prefer if Plex didn't have to re-identify everything. EDIT 2: I'm fairly sure it has something to do with the cache, as nothing happens when I invoke the mover, and there is a docker.img on the cache. EDIT 3: Actually, the mover does move stuff that is in my share. But there is still a 20 GB docker.img on there that won't move. EDIT 4: I used templates to re-create my Plex and PlexPy docker, and all the content is there so it's all good. I'm not sure what to do with the 2 docker.img's though. EDIT 5 (sorry): I followed the instructions here on how to delete your docker.img, and it disappeared from my cache drive. But the one on Disk 1 remained, and when I went to the Docker tab, there everything was, just as it was before everything went down. So I think I'm good now! Edited November 1, 20178 yr by nadbmal
Archived
This topic is now archived and is closed to further replies.