May 17, 20179 yr Hey guys, I'm in a state of panic at the moment, I have no idea what is going on..... So last night, I was watching TV via Plex, so all was fine. This morning I wake up and notice one of my docker WEBUI's is no longer responding. I went to my unraid gui and checked the log, but nothing was writing of value, so I thought the docker image may just be stuck. I took the array offline rebooted. Once rebooted disk1 was unmountable, so I put the array in maintenance mode and ran xfs_repair on it, however it wouldn't let me do anything without destroying the logs on the disk and as I could not mount the disk to fix it, I ran xfs_repair -L which seemed to work properly as I can now mount the drive. However, disk one shows NO data anymore Screenshots below of Disk 1, also diagnostics attached This has also causes all my dockers to dissappear, all my shares to dissappear... I see data on the other drives, but I don't want to go forward without some advice on what to do next... Any idea's? Did I fuck up royally here :(? unraid-diagnostics-20170517-1050.zip
May 17, 20179 yr Go to docker settings and see if you appdata location is valid first. Reason you can't see it likely is the docker.img file was corrupted or not found. Do you only have the 2 drives? If so your parity is actually a raid 1 mirror rather than just a non file system parity calculation. Whatever you do any parity rebuild until you confirm configuration with the community prior to problem. This includes drive counts. I don't want you overwriting parity/mirror data if nothing is seen on your drive. Edited May 17, 20179 yr by phbigred
May 17, 20179 yr Author 2 minutes ago, phbigred said: Go to docker settings and see if you appdata location is valid first. Reason you can't see it likely is the docker.img file was corrupted or not found. Do you only have the 2 drives? If so your parity is actually a raid 1 mirror rather than just a non file system parity calculation. Whatever you do do not start the array until you confirm configuration with the community prior to problem. This includes drive counts. I don't want you overwriting parity/mirror data if nothing is seen on your drive. Hi phbigred, No I have 5 disks and 1 Parity drive, the data on the other disk seem fine This is what im getting in the docker settings I'm not sure what to do to get disk 1 to show its data again Maybe I should remove the disk and get a new one and see if it rebuilds? -A
May 17, 20179 yr What do the shares show? My guess is appdata lived on disk 1. What does MC show from putty or unraid boot to GUI? Midnight commander you'd back out to /mnt/disk1. Looks like old school dos shell. Take a peek in there. Who knows your data data may be there but the state it was in when doing the xfs repair might need a reboot to reset the shares back. Report back on what shows in /mnt/disk1 first before going that route. Based on your info will push us one direction or rebuild of disk1. Edited May 17, 20179 yr by phbigred
May 17, 20179 yr Author Shares: If I attempt to make a new share, it just says "Share deleted" This is what MC shows: If I browse ?disk1, it shows nothing
May 17, 20179 yr Parity is showing green which is a good sign. If you want to try you can rebuild your drive, or get a new drive. Before you do either, shut down and reseat cables on disk 1 and on motherboard. I'll have to defer to Turl or someone for a deep dive into diagnostics. Let's hold on steps right now for recovery and verify cabling. Grab one more diag and save it somewhere. Stop the array, change setting auto start array to no in settings under disk settings and shutdown and reseat cabling and turn back on. Then we hold... The disk being green balled on disk 1 makes me hesitant on a rebuild from parity right now. Edited May 17, 20179 yr by phbigred
May 17, 20179 yr My gut instinct is to disconnect disk1, start the array degraded and see if parity actually agrees with what's on the physical disk1. At the very least you would then have 2 copies of the data to try a recovery. Ideally you could then rebuild to a new drive to get the array back protected, and try data retrieval on the removed disk. Do you have backups? How much time and money is the data on disk1 worth?
May 17, 20179 yr Author 1 hour ago, jonathanm said: My gut instinct is to disconnect disk1, start the array degraded and see if parity actually agrees with what's on the physical disk1. At the very least you would then have 2 copies of the data to try a recovery. Ideally you could then rebuild to a new drive to get the array back protected, and try data retrieval on the removed disk. Do you have backups? How much time and money is the data on disk1 worth? Hi Jonathanm, I don't have backups, I use this unraid basically just for torrents and resilio, which syncs data from other NAS drives I look after, so loosing data isnt a big deal for me, obviously a bummer but not a deal breaker. So best course of action you guys think is to grab another drive and attempt a rebuild? @phbigred I checked all the cables, and put a new sata cable on disk1, unfortunately no change
May 17, 20179 yr Yeah do what Jonathan mentioned. Recommend a new drive and process laid out above. Worst case you may be able to recover the missing data outside the array.
May 17, 20179 yr Author 1 minute ago, phbigred said: Yeah do what Jonathan mentioned. Recommend a new drive and process laid out above. Worst case you may be able to recover the missing data outside the array. @jonathanm Okay for now (until I get a new disk) i have removed disk 1 from the array Shares and dockers are now working, so looks like the parity is fine. I'll get a new drive delivered in the next few days and do a rebuild. Thanks for the help lads, much appreciated +rep all around
May 17, 20179 yr Community Expert If you haven't rebooted since disabling disk1 please grab and post new diags, I'm curious to seem them since the only problem on you previous diags is remaining file system corruption on disk1: Quote May 17 10:47:15 unRAID kernel: XFS (md1): Mounting V5 Filesystem May 17 10:47:15 unRAID kernel: XFS (md1): Starting recovery (logdev: internal) May 17 10:47:16 unRAID kernel: XFS (md1): Metadata corruption detected at xfs_inobt_read_verify+0x40/0x43, xfs_inobt block 0x3c4198e8 May 17 10:47:16 unRAID kernel: XFS (md1): Unmount and run xfs_repair May 17 10:47:16 unRAID kernel: XFS (md1): First 64 bytes of corrupted metadata buffer: May 17 10:47:16 unRAID kernel: ffff880207ed7000: 41 42 33 43 00 00 00 fd 00 43 c8 c9 00 43 db 9a AB3C.....C...C.. May 17 10:47:16 unRAID kernel: ffff880207ed7010: 00 00 00 00 3c 41 98 e8 00 00 00 41 00 07 6b e5 ....<A.....A..k. May 17 10:47:16 unRAID kernel: ffff880207ed7020: 33 bb 41 75 1c 3a 44 a8 a8 7c 0b e3 23 66 c9 1d 3.Au.:D..|..#f.. May 17 10:47:16 unRAID kernel: ffff880207ed7030: 00 00 00 02 af 8f b6 4e 01 c9 93 41 00 00 00 01 .......N...A.... May 17 10:47:16 unRAID kernel: XFS (md1): metadata I/O error: block 0x3c4198e8 ("xfs_trans_read_buf_map") error 117 numblks 8 May 17 10:47:16 unRAID kernel: XFS (md1): xfs_do_force_shutdown(0x1) called from line 315 of file fs/xfs/xfs_trans_buf.c. Return address = 0xffffffff812c0558 May 17 10:47:16 unRAID kernel: XFS (md1): I/O Error Detected. Shutting down filesystem May 17 10:47:16 unRAID kernel: XFS (md1): Please umount the filesystem and rectify the problem(s) May 17 10:47:16 unRAID kernel: XFS (md1): xfs_difree_inobt: xfs_inobt_lookup() returned error -117. May 17 10:47:16 unRAID kernel: XFS (md1): Ending recovery (logdev: internal) Running xfs_repair again on disk1 should have fix it, or maybe just unmounting and remounting would be enough, hence why I'd like to see the new diags.
May 17, 20179 yr 9 hours ago, Addy said: Okay for now (until I get a new disk) i have removed disk 1 from the array Keep in mind if another drive fails you will lose what is still showing on slot 1 and whichever other drive fails. Right now with the physical drive missing slot 1 is using ALL your remaining drives to appear normal.
May 18, 20179 yr Author 14 hours ago, johnnie.black said: If you haven't rebooted since disabling disk1 please grab and post new diags, I'm curious to seem them since the only problem on you previous diags is remaining file system corruption on disk1: Running xfs_repair again on disk1 should have fix it, or maybe just unmounting and remounting would be enough, hence why I'd like to see the new diags. Hey Johnnie, Here are the latest Diagnostics, I have not rebooted since removing disk1 from the array (software wise) with XFS_Repair, do I need to run xfs_repair again after destroying the logs with -L? -A unraid-diagnostics-20170518-1141.zip @jonathanm I have a new drive on its way ps, sorry for the late reply, aussie time is poo Edited May 18, 20179 yr by Addy
May 18, 20179 yr Community Expert 6 hours ago, Addy said: with XFS_Repair, do I need to run xfs_repair again after destroying the logs with -L? Usually not, but in this case it was needed. Emulated disk mounted, this would imply parity was not 100% in sync, either way since the disk looks fine you can rebuild to the same disk and if all goes well keep the new one as spare, so you spend the least amount of time possible with the array unprotected. Edited May 18, 20179 yr by johnnie.black
May 22, 20179 yr Author On 2017-5-18 at 6:16 PM, johnnie.black said: Usually not, but in this case it was needed. Emulated disk mounted, this would imply parity was not 100% in sync, either way since the disk looks fine you can rebuild to the same disk and if all goes well keep the new one as spare, so you spend the least amount of time possible with the array unprotected. Ended up putting the new disk in, as it is larger than the one we were having issues with. Since you think its all good, I'll format it and label it as a spare for now Thanks for your help
Archived
This topic is now archived and is closed to further replies.