dogfluffy

And the saga continues. Got my ethernet dongle and now I have another red ball, disk2. I pulled a smart report. I also connected it via different power and SATA, booted fresh and got the same results. DISK_DSBL in unMenu. Did I just have another disk fail? Disk2Hitachi.txt

It finished. I got the same results from windows apps and chromebook app as before. Anyway I already ordered a USB to gigabit dongle so that's probably what I'll do to clean up the directories and remove the lost+found entirely. Then clean upgrade to unraid 6.

Nice. And since my linux is dangerous LOL. newperms /mnt/user/lost+found is the proper argument? It's a share spanning disk9,disk10.

I'm still on 5.05 version, but it looks like the New Permissions utility is global and not able to run it against a single folder. Would this be a good time to upgrade to 6?

This was kind of interesting. I was first starting with the lost+found, then wanted to see what this chromebook could do, since it's already my 1st laziest option. Found @ Play store CX File Explorer and eventually connected to the lost+found and browsed around. My top dir is 2_4 and filled with data in the app. On my windows 7 machine I couldn't view 2_4. I also for fun downloaded Midnight Commander for windows, although I'm sure you mean to install it on my Unraid machine or just another linux machine? At any rate the MC in windows couldn't open 2_4 either. It was just dumb luck I knew there was actually data there. The file copy speed is awful due to wireless, but I'm sure I can rig something up to clean up this folder structure...possibly even in a future project. I kinda wanted to play with a Pi or I could easily install linux real fast on some old hardware with gigabit ethernet ports already. I'm assuming native linux file systems are why it succeeds?

It only seems to when you have the array stopped, that I could figure out anyway. In the Disk Management tab, unassigned disks will have a mount button. unMenu does have a file browser tab for the mounted disks where I think it would display but also just browsable normally I guess once mounted. I'm sure I could console command it too with some Google. I think I still have an Ext3 reader on my Windows 7 box and a USB Sata adapter I was using so many years ago. I also have 2 drives with identical serial numbers, but one is n and m. I would bet I mixed those up along the way. Going to fix with a sharpie. I just stop maintenance mode and start up normal now right?

Not yet, but I watched the console scroll off and on. Looks promising! I didn't want to screw it up now by mounting something wrong, although I think we're basically finished now? I think that before I thought the parity works backwards but now I understand it. I also have the 2 disks I pulled out along the way. So once I restart the array I guess I can copy it back over? I also had a weird counter intuitive disk assignment for my data share, where I always assigned new disks as available to the share, thinking it would always be large enough. But it sort of scattered some data it seems across the nearly blank disks too. I'm still piecing it back together but it looks good and again no errors on the unraid main screen. I suppose I may have had a flaky controller or loose power, or all 3 at once going on. Thanks for all your help with this, and teaching me how it actually works. It's parity magic.

OK it finished. Has a long console message, but Objects w/o names 2692, Empty dirs 25, Dirs linked /lost+found: 235, Dirs w/o:25,Files linked to lost+found 2457, pass 4 finisheddone 576403, 603/sec, Deleted unreachable 53670, flushing, syncing finished. Back to prompt.

Wall that was my second guess. Thanks for the explanation! I *think* this might explain why disk13 and disk3 (I believe) had very similar data? It also could be just from my mistake disk shuffling without proper labels.

It's working! Counting down from a very large number. I'm going to grab some lunch. Thanks again for the help. I assume this is processing the lost+found folder or does it run off parity? I'm not sure what handles lost+found?

Hah yeah I swear I hit the button! Anyway ran and got Bad root block 824657253. (--rebuild-tree did not complete) Aborted (cored dumped)

Thanks to advances in modern logistics I'm already back up and running! I still have some housekeeping to do hardware wise to clean all this up, but new drive and controller are installed and running. I assigned the new disk13 and rebuilt last night with 0 errors. So I'm back to the Unformatted Disk10 and an essentially blank Disk11. I'm not sure how to best proceed, but I can start viewing files and trying to piece together what data is missing, and what data I have on previous unraid drives. There was one console message about a reiserfs error on device md10(disk10), probably due to my scan of that disk I gather, but no messages since and no bad noises. Also I forgot about a lost+found on disk9. This is probably the useful bit of info you need? Feb 4 16:34:20 Calculon logger: mount: /dev/md10: can't read superblock Feb 4 16:34:20 Calculon emhttp: _shcmd: shcmd (42): exit status: 32 (Other emhttp) Feb 4 16:34:20 Calculon emhttp: disk10 mount error: 32 (Errors) Feb 4 16:34:20 Calculon emhttp: shcmd (43): rmdir /mnt/disk10 (Other emhttp) Feb 4 16:34:20 Calculon emhttp: shcmd (44): mkdir /mnt/disk11 (Routine) Feb 4 16:34:20 Calculon emhttp: shcmd (45): set -o pipefail ; mount -t reiserfs -o user_xattr,acl,noatime,nodiratime /dev/md11 /mnt/disk11 |$stuff$ logger (Other emhttp) Feb 4 16:34:20 Calculon kernel: REISERFS error (device md10): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [1 2 0x0 SD] (Errors) Feb 4 16:34:20 Calculon kernel: REISERFS (device md10): Remounting filesystem read-only (Drive related) Feb 4 16:34:20 Calculon kernel: REISERFS (device md10): Using r5 hash to sort names (Routine) Do you need any logs or more info from me?

Thanks again for all your help. It looks like shipping is ETA Feb 6, and I should be set to bring the array online and rebuild it, again. Hopefully no gremlins this time and be able to see what the file situation looks like.

Excellent. Yes I have a project now. I ordered a fresh drive and I'll label these to the unraid serial number and re-inspect these power connections before I put it all back together. If I bring it all online with the new Supermicro controller and the new drive ready to become disk13 during a rebuild is that the optimal plan going forward next week hopefully?

https://www.rosewill.com/product/rosewill-rsv-l4500-4u-rackmount-server-case-or-chassis-15-internal-bays-8-cooling-fans-included/ Oh no this thing is all ripped apart right now. I disconnected the drives attached physically to one of the onboard controllers and also a string of SATA power on a molex splitter. It's a mess, but I was just trying to get a SMART report off disk13. I thought it was new, but is apparently trashed regardless. I have 2 other 3TB drives but they have files on them I may need, I don't know yet. It's sounding like this may be trashed...but I could order another 3TB and I already ordered the Supermicro controller. It might be best to just let this sit and try to isolate what is causing the bad mojo. Those 3 disks were connected to the same controller with the 4th cable unused, and I also had a whole series of wyes and power adapters going on I couldn't see very well along the fan rail. I can't tell how many because I think I needed a couple just as extensions for a longer run and it was all tucked away out of sight.

And yeah, disk13 I connected to the supermicro controller and a native SATA power connection and it shows DISK_DISABLE_NP in unMENU and isn't even available in the disk management pulldown to try and run SMART report on. I also had it connected to another wye on accident and rebooted with the intended native SATA and now it just says DSK_DSBL in unMENU and not installed on the normal unraid screen. It still won't run smartctl and returns failed. So I suppose I should order another disk? I had been thinking about buying a 6 or 8 TB to be my next parity drive down the road, if I was going to upgrade, someday. It's looking like someday is here. Statistics for /dev/sdd ST3000DM001-1CH166_W1F2JKX7 smartctl -a -d ata /dev/sdd smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ Smartctl: Device Read Identity Failed (not an ATA/ATAPI device) A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

Gah lag. Yes in every sense, I think this is it. I had a power molex connecting to a string of SATA power connections and it looks like it might have wiggled loose and been causing problems. I ALSO seem to have a bad drive, although that could be the result of this loose connection. I pulled disk13 and connected it via a USB with it's own power and it says Smart read failed and was disabled unless via ATA connection. I had it connected via the ATA and it would report that SMART was disabled. I tried to enable it from the console but was unsuccessful. I can connect it back up to a different power rail and reboot inside the case again. I think this was the drive that was clicking though. I was able to get the SMART off disk8 and disk10 and attached the report. Disk13 is still connected via the USB right now. Of course 5 minutes before I ordered the Supermicro controller card. I should try to find direct SATA power cables for my Seasonic modular power supply so I can toss these power Molex to SATA wyes. smartreportpartial.zip

Ok I managed to get some of the drives identified and also physically ID'd the bad controller. At this point if I disconnect the 4 drives that are connected to the other onboard controller and connect these last 3 drives. I could then pull the SMART logs and it wouldn't hurt anything or cause any more errors since the bad controller isn't connected to anything? I connected 8,10,13 to the other onboard SATA controller and rebooted but they didn't appear in unraid at all. Any idea what is going on here?

Yeah I don't know but I'm assuming one of the onboard controllers crashed and died during the rebuild last night. I had no errors @ around 50% last I checked before I crashed. Console was spamming and locked. I rebooted it, missing 4 disks. I'm also thinking it was an intermittently failing controller this whole time. There maybe also be a failing disk connected to a controller and that is the clicking I hear. During the rebuilding yesterday it was humming along with no noises or errors so I've been a bit panicked and distracted this morning to find out what was going on. I only had to pull a few disks to ID the Western Digital so the drives aren't labeled and it's a tight fit to pop the rack out, unlatched the drive rail and pull to read the serial number to label it. I just wanted to get the controller card shipping today if 2 of those controlling the disks is a viable solution and not causing a bottleneck or other issue.

No you're fine it's just text, I didn't know I could do any of that with my chromebook. I couldn't find a save as, only copy to. Anyway I think I have a failing controller. I can also hear an odd clicking on one of the drives, not like a failure click as much as like something polling it, maybe. I was actually just now thinking I should look into another one of the Supermicro pci controller cards. Assuming it's say $100 ish and would ultimately solve my problem here. I was just looking over the setup, and it's a nice server board, low power draw, stable and ECC ram so an add-in controller might bypass the onboard controller/driver issues and provide a clean Ver 6 Unraid upgrade path once this mess is sorted out. When I had previously attempted to upgrade Unraid I was missing 4 disks upon booting up, so I rolled it back to my backup. There are 2 different onboard 4 port SATA controllers. Supermicro Sas2lp-mv8 is the card I was looking at for $50 off ebay. I have one already installed.

I'm on a chromebook. Is there some problem with these logs? It was a google doc. I exported it as .txt and found a zip utility. Archive.zip

syslog-2019-01-31.txt Smart report.gdoc

OK I ran it through a clean cycle of memtestx86 just to be sure and it passed. I keep getting messages about this ATA6 link is slow, waiting and resetting at is boots up. Now I'm to the console login screen. I can reboot and try to copy verbatim if this isn't already captured. How should I proceed?

I can't really do anything due to the console spamming the error. I tried to stop the array to shut down and bring up clean to pull your logs but it is hung up, scrolling across the bottom of the unraid webpage that error unmounting share(s) and retrying unmount.... It's kind of locked up. Also does the unMENU have a syslog tool? I've only been too busy to actually do it, but I'm here trying to recover my data so yes, it is important to me.