SlowNIC

It appears that only 13 files were affected out of 705. There is no way I could recover this data without a ddrescue or a similar tool. Given the amount of time that unraid V6 has been released, I don't understand why I am the first person to run into this, but I am grateful to the community for their generous support in getting it resolved. root@Tower1:~# echo -n 'UnRaid 6 with ddrescue ' >~/fill.txt root@Tower1:~# ddrescue --force --fill=- ~/fill.txt /dev/disk/by-id/ata-ST33000651NS_Z292V792-part1 /boot/WCAZA5082257.ddrescue GNU ddrescue 1.21 Press Ctrl-C to interrupt Initial status (read from mapfile) filled size: 0 B, filled areas: 0 remaining size: 6385 kB, remaining areas: 1016 Current status filled size: 6385 kB, filled areas: 1016, current rate: 0 B/s remain size: 0 B, remain areas: 0, average rate: 0 B/s current pos: 143929 MB, run time: 0s Finished root@Tower1:~# xfs_repair /dev/disk/by-id/ata-ST33000651NS_Z292V792-part1 xfs_repair /dev/disk/by-id/ata-ST33000651NS_Z292V792-part1 Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done root@Tower1:~# mkdir /mnt/test root@Tower1:~# mount /dev/disk/by-id/ata-ST33000651NS_Z292V792-part1 /mnt/test -o ro root@Tower1:~# find /mnt/test -type f -exec grep --binary --quiet 'UnRaid 6 with ddrescue ' {} \; -print /mnt/test/media/myth/1378_20150519113400.mpg /mnt/test/media/myth/10997_20140807035900.mpg /mnt/test/media/myth/11039_20130619035900.mpg /mnt/test/media/myth/11012_20140528125900.mpg /mnt/test/media/myth/11095_20140616040300.mpg /mnt/test/media/myth/1386_20140401144400.mpg /mnt/test/media/myth/10992_20151026052900.mpg /mnt/test/media/myth/3441_20160603005900.mpg /mnt/test/media/myth/11028_20170629053200.mpg /mnt/test/media/myth/3041_20141124022900.mpg /mnt/test/media/myth/11040_20131012065900.mpg /mnt/test/media/myth/11079_20151114050100.mpg /mnt/test/media/myth/1195_20170319160000.mpg 1378_20150519113400.mpg INDIEHD*378*dish2 May 19 09:34 'Water' 10997_20140807035900.mpg SHOWHD*9460*dish6 Aug 07 01:29 'Blood Out' 11039_20130619035900.mpg TLCHD*9488*dish4 Jun 19 01:01 'My Teen Is Pregnant and So Am I @ For the Love of Family' 11012_20140528125900.mpg TNTHD*9420*dish5 May 28 10:01 'Supernatural @ Plucky Pennywhistle's Magical Menagerie' 11095_20140616040300.mpg CCHD*9485*dish7 Jun 16 00:36 'South Park @ Cartman Finds Love' 1386_20140401144400.mpg SONYHD*9529*dish1 Apr 01 12:39 'The Body' 10992_20151026052900.mpg WCVBDT*6318*dish5 Oct 26 02:31 'Elementary @ M.' 3441_20160603005900.mpg WGBXDT*43*UHF Jun 02 22:01 'Miss Fisher's Murder Mysteries @ Blood and Money' 11028_20170629053200.mpg AETVHD*9419*dish2 Jun 29 02:05 'Storage Wars: Northern Treasures @ The Seven Habits of Highly Effective Instigators' 3041_20141124022900.mpg WBZDT*30*UHF Nov 23 22:31 'The Good Wife @ The Trial' 11040_20131012065900.mpg APLHD*184*dish2 Oct 12 04:01 'Monsters Inside Me @ It Came From a Tick ...' 11079_20151114050100.mpg LIFEHD*9470*dish7 Nov 14 01:03 'Step It Up @ Fan Chat: Beware of Bat Woman' 1195_20170319160000.mpg AHC*195*dish6 Mar 19 13:01 'Dawn of the Apocalypse @ Dark Age of Egypt'

First pass completed, now I will identify the files that were affected... root@Tower1:~# ddrescue --force /dev/disk/by-id/ata-WDC_WD20EARS-00MVWB0_WD-WCAZA5082257-part1 /dev/disk/by-id/ata-ST33000651NS_Z292V792-part1 /boot/WCAZA5082257.ddrescue GNU ddrescue 1.21 Press Ctrl-C to interrupt ipos: 143929 MB, non-trimmed: 0 B, current rate: 14848 B/s opos: 143929 MB, non-scraped: 0 B, average rate: 11819 kB/s non-tried: 0 B, errsize: 6385 kB, run time: 1d 23h 39s rescued: 2000 GB, errors: 1016, remaining time: 0s percent rescued: 99.99% time since last successful read: 0s Finished root@Tower1:~# WCAZA5082257.ddrescue

ddrescue has been added to "Nerd Tools". Thank you to all who made this happen.

For those of us who have broken arrays (parity invalid) and need to recover data disks there is "ddrescue". It was available in unraid V5 through unmenu. I'm not aware of it being available for unraid V6. It is critical that this is available when all other data recovery options fail. Could this please go into nerd pack? wget http://slackware.cs.utah.edu/pub/slackware/slackware64-14.2/slackware64/ap/ddrescue-1.21-x86_64-1.txz installpkg ddrescue-1.21-x86_64-1.txz

Thank you so much! Copy is underway so far with a 26 hour ETA. I will make a request in Nerd plugin thread. Woo Hoo!

Thanks again for the replies. I was aware of the dd with the options. It is a good suggestion, but it is extremely time consuming as it does not track which blocks it was unable to read. You have to keep track manually with a script and retry them. That was why they wrote ddrescue. I've had this happen before, and using ddrescue, I've been able to recover 99% of my data from a drive in this condition. Perhaps I lose 3 files. The file integrity plug in tells me exactly which ones were affected. I've been doing this for 12 years, but feel free to remind me about lessons anytime if you think next time is the time it will "sink in" :-)

I do appreciate the responses. The array is broken. What do I mean by broken? The parity drive is not valid. How did it become invalid? The question is not relevant for this thread. I may start a new thread to address that. So I have a known data drive with issues and an invalid parity drive. I've already done the hardware troubleshooting and confirmed that 3 different ways. The 4th way to confirm it is the smart report that I attached, below... So my concise question is, with a broken array, and a data drive with errors, what do people do to recover data? Up to this point, I thought ddrescue was the only option, yet ddrescue is missing from unraid V6, something that I view as a basic tool for maintaining storage subsystems when other measures fail. I must be missing something, because I know that people do have the combination of "broken array" plus "drive with errors" happen. Nobody is denying that this happens, right? So, do people fire up some virtual machine solution that runs ddrescue? Do they boot another flash drive that runs ddrescue? Is there a different utility that efficiently reads and handles errors like ddrescue does? Some other solution that I am missing? WDC_WD20EARS-00MVWB0_WD-WCAZA5082257-20180113-1433.txt

Parity rebuild (reading the disk I am trying to recover) is taking 75 days and produces thousands of recoverable errors on the disk. I have 6 arrays of 24 disks each, so this does happen to me from time to time. This is the first time with version 6 and I find myself without one of my most basic tools that I have relied on for years. I feel like one arm is tied behind my back. So, I figure I must be doing something different from everyone else to have this problem. I'm not the first one that needs to recover a disk, right? So, my version 6 solution now seems to be to remove the drives from the array and find a linux system that has ddrescue. It is not as nice as my version 5 solution, which was to type ddrescue at the command prompt, but it still can work. What do other people do?

Thank you for the suggestion. Follow up question to the group: Is this the method that people use to recover data from a failed disk from a broken array? The drive is readable, but the rebuild parity operation on the array was taking 75 days and making thousands of recoverable errors. Prior experience tells me that rebuild would fail prior to the 75 day window completing. Prior experience also tells me that ddrescue would recover most of my data quickly and efficiently. Dynamix File Integrity plugin that I run on all my XFS drives would then allow me to identify any corrupted files. How are people recovering from data loss in version 6 without ddrescue? Or am I asking the question wrong? Sorry my mind is stuck back in version 5 when this was easy to my old way of thinking.

I've searched the forums for about an hour now and I give up, sorry for bothering everyone. I had a handle on it in V5. Can't find "ddrescue" tool anywhere, closest I came was Nerd tools plugin. If I don't get any responses, I'll pull the drives and plug them into another computer that runs linux. Thanks in advance, and sorry for my poor searching skills (this must have been asked before).

Another user chiming in on the LSI mptsas spinup issue. Thanks.

One day only... * Western Digital Caviar Green WD15EADS 1.5TB SATA 3.0Gb/s 3.5" Internal Hard Drive - OEM * Limit 10 per customer * Shipping - The shipping cost for this product is $0.00 (Free) via UPS shipping (guaranteed 3 business day service). http://www.newegg.com/Special/ShellShocker.aspx

Unexpected surprise: They have a 3 year and 3 month warranty! Warranty: Serial Number Seagate Part Number Warranty Status 9VS062VT 9JU138-302 In Warranty Expiration 09-Oct-2013 All six came today in UPS ground. Factory refurbished package (double static bagged w/dessicant) Labels on drives have the standard "Seagate Certified Repaired Hard Drive" for refurbs... latest firmware.... Looks like MacMall came through for me... I know - I was lucky...

According to the vendor, I have 6 coming for Tuesday. I was not aware of the bad reviews for this vendor, thank you for pointing it out. I know that 33 drives is not a statistically significant sample, and I know that I've only been running half of them for about a year, but the Seagate 1.5TB "refurbished" drives have been kind to me (no failures yet). Perhaps it is because I run them using Unraid, 30 minute spin-down, and media server application. I can say the same for the 21 WD10EACS (1TB Western Digital Green) drives that I have. 7 of those are refurbished. 30 days or more warranty is all I consider a firm requirement. More warranty is worth some amount of money, the exact amount to be decided by the particular individual making the purchase and which side of the brain they are thinking with... I did have a 500GB Western Digital fail after 2 years (new) and a 750GB Seagate fail after 1.25 years (refurb), but those are my only disk failures in 3 years of Unraid use. I have a total of 68 drives. Maybe I'm just lucky and I have yet to learn an expensive lesson, but from a $$$ point of view, refurbished has allowed me to take my media server obsession to new heights.

I know that some folks have crunched the numbers and that the 2TB drives seem to work for them. So, I'd thought I'd share this deal in case someone is interested. For my needs, I have not been able to build a formula that shows these drives as cost effective versus the 1.5TB. I'm watching this great drive waiting for the price to come down further. From slickdeals.net http://accessories.us.dell.com/sna/products/System_Drives/productdetail.aspx?c=us&l=en&s=bsd&cs=14&sku=A2488823 Dell SB has 2TB Western Digital Caviar Green SATA Hard Drive for $199.99 - $20 with code XTHR0RQRXJHQ12 = $179.99 with free shipping. Next lowest on Google Products is $230 shipped. Thanks linxcod & ShadowGuard

As far as how important warranty is, that depends on a number of factors... I'm assuming that everyone here is filling up their Unraid box(es) and that the cost per SATA port is about $40 with hardware and unraid license. So, by the time you get to the third year of a warranty, the drive sizes have become so much larger, that you probably would not want to spend the time packing and the money shipping the broken drive for warranty when the drive size has doubled or tripled. After all, that "mini-drive" is pulling electricity and not providing as much capacity. It's using up that $40 SATA port on your Unraid box. Time to upgrade anyway. This is what I am finding in my own experience. I don't even bother sending in that broken 500GB drive that still has 3 months warranty left. I could buy a faster one that uses 60% less electricity for $40 (refurbished) if I really needed it, but I do not because the price point now is 1500GB. Add in the fact that statistically, drive failure is unlikely after the first month provided you have proper power and cooling and I'd say that the warranty is worth about $10 to me personally. Depending on your situation and experience, you may have a different opinion. If this is the only drive you buy, get the warranty. Otherwise, buy 10 of them and if one of them fails after 15 months you are still ahead of the game. Especially because Unraid saved your data!

I'm assuming it is the 90 day warranty and that these come from the factory with the new firmware. Otherwise, if not from the factory, it would probably be what remains on the 3 year and possibly old firmwire. Those two are the most common when I get a refurbished Seagate drive, but I have not received any from this particular company yet. Their website leaves you with the impression that these are "factory refurbished", but it looks like a text box that they paste on any of the products that have the word "refurbished" in the title. To prove this point, first look at the item in question (the drive), then go to the search bar and type in "refurbished" for the search. Click on any of the products that come up and you'll see the same "manufacturer refurbished" blurb on each of the products.

These have been reliable for me after the firmware upgrade procedure. I have 33 of these drives connected to Unraid version 4.4.2. http://www.macmall.com/ttsvr/p/5368796?dpno=7766570 There have been plenty of people who will tell you to stay away from these drives. Make your own decision, but I believe this is a good price that is worth mentioning.

I'm getting a Kernel OOPS with more than 16 drives. I had an array that I've been using for 4 months. It has 16 drives of the same model number. I added an additional 4 drives of the same model number. Here are my steps: 1. Add the 4 drives, but only connect the power cords, not the SATA cables. Test for enough power and cooling. Ran for two days without issues. 2. Connect one of the new drives to a spare port on the mainboard. This port is on the Promise PDC42819 and we're using AHCI. Boot-up add the drive in the "drives" screen as disk16, back to main, "restore", "start" - generate parity. As soon as I hit start, I get the Kernel OOPS. 3. Undo step 2 and move some of the existing drives to fully populate the Promise PDC42819 ports on the mainboard. We're still at 16 drives - go to the devices screen, shuffle accordingly, remove disk16. Back to main, "restore", "start" - generate parity. Works. Wait for parity sync to complete. Watch movies... 4. Next day - connect a different new drive to a spare port on the Adaptec 1430sa controller. We go into devices screen, add disk16, and back to main, restore and start - Kernel OOPS. 5. Undo step 4. Move some existing drives from the sata_sil24 controllers to fully populate the Adaptec 1430sa, reconfigure for 16 drives, start - everything works, generate parity, watch movies.... 6. Next day - connect a different new drive to a spare port on the sata_sil24 controller. We go into the devices screen, add disk16, and back to main, restore and start - Kernel OOPS. 7. Undo step 6. Move some existing drives from the SB600 controller on the mainboard to fully populate the sata_sil24 controllers, reconfigure for 16 drives, start - everything works, generate parity, watch movies..... 8 Next day - connect a different new drive to a spare port on the SB600 mainboard controller. We go into the devices screen, add disk16, and back to main, restore and start - Kernel OOPS. Get the idea? The point is that these controllers have been happy in my system for months. I've had the 16 disks distributed among these controllers so that I would balance the load. No matter which controller I add disk16 (the 17th disk) to, the Kernel OOPS happens. It is independent of which new disk I pick or which controller I add it to. I'd bet that there is still some data structure somewhere that has a 16 disk limit and that I'm trashing memory. It's visible on my system due to the combination of drivers, perhaps? I had one of the OOPS in an emacs buffer on another system. Most of the time the OOPS will crash the system, but the one time it didn't I captured it in the emacs buffer. Unfortunately, I did not save the buffer before I did a reboot of that system. unRAID Server Pro 4.5-beta6 Linux Tower4 2.6.29.1-unRAID #2 SMP Thu Apr 23 14:17:18 MDT 2009 i686 AMD Athlon X2 Dual Core Processor BE-2400 AuthenticAMD GNU/Linux sata_sil24 (4 controllers, 2 ports each, 8 ports total) sata_mv (adaptec 1430sa, 1 controller, 4 ports each, 4 ports total) sata_ahci (promise PDC42819, SB600, 2 controllers, 4 ports each, 8 ports total) atiixp (single IDE cache disk) Other than capturing more data, is there some procedure I can try to get a smoking gun? Thanks in advance.

It looks like the 2.6.29.1 Linux kernel still cannot do HDIO_GET_IDENTITY on a SATA drive on my SuperMicro LSISAS1068E SAS controller. Not an unraid problem, but to my simple understanding this might not bode well for using SuperMicro SAS controllers with unraid. parity: pci-0000:00:12.0-scsi-1:0:0:0 (sdd) ata-WDC_WD10EACS-00ZJB0_WD-WCASJ1908386 disk1: pci-0000:02:00.0-sas-0x5003048000610e80:1:0-0x1221000004000000:4 (sdb) scsi-SATA_WDC_WD10EACS-00_WD-WCASJ0766868 disk2: pci-0000:03:00.0-scsi-1:0:0:0 (sdc) ata-WDC_WD10EACS-00D6B0_WD-WCAU40627966 I can add the device on the "Devices" page, but when you go back to "Main": parity: pci-0000:00:12.0-scsi-1:0:0:0 (sdd) ata-WDC_WD10EACS-00ZJB0_WD-WCASJ1908386 disk1: not installed disk2: pci-0000:03:00.0-scsi-1:0:0:0 (sdc) ata-WDC_WD10EACS-00D6B0_WD-WCAU40627966 # uname -a Linux Tower5 2.6.29.1-unRAID #2 SMP Thu Apr 23 14:17:18 MDT 2009 i686 AMD Athlon X2 Dual Core Processor BE-2400 AuthenticAMD GNU/Linux # dmesg Fusion MPT base driver 3.04.07 Copyright © 1999-2008 LSI Corporation Fusion MPT SAS Host driver 3.04.07 mptsas 0000:02:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18 mptbase: ioc0: Initiating bringup ioc0: LSISAS1068E B3: Capabilities={Initiator} mptsas 0000:02:00.0: setting latency timer to 64 scsi1 : ioc0: LSISAS1068E B3, FwRev=011a0000h, Ports=1, MaxQ=266, IRQ=18 scsi 1:0:0:0: Direct-Access ATA WDC WD10EACS-00Z 1B01 PQ: 0 ANSI: 5 sd 1:0:0:0: [sdb] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB) sd 1:0:0:0: [sdb] Write Protect is off sd 1:0:0:0: [sdb] Mode Sense: 73 00 00 08 sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sd 1:0:0:0: [sdb] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB) sd 1:0:0:0: [sdb] Write Protect is off sd 1:0:0:0: [sdb] Mode Sense: 73 00 00 08 sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sdb: sdb1 sd 1:0:0:0: [sdb] Attached SCSI disk libata version 3.00 loaded. md: initializing superblock md: import disk0: HDIO_GET_IDENTITY ioctl error: -22 md: import disk0: HDIO_GET_IDENTITY ioctl error: -22 md: unRAID driver removed md: unRAID driver 0.95.2 installed md: xor using function: p5_mmx (7898.800 MB/sec) md: import disk0: [8,48] (sdd) WDC WD10EACS-00ZJB0 WD-WCASJ1908386 offset: 63 size: 976762552 md: import disk1: HDIO_GET_IDENTITY ioctl error: -22 md: import disk2: [8,32] (sdc) WDC WD10EACS-00D6B0 WD-WCAU40627966 offset: 63 size: 976762552 md: import disk0: [8,48] (sdd) WDC WD10EACS-00ZJB0 WD-WCASJ1908386 offset: 63 size: 976762552 md: import disk1: HDIO_GET_IDENTITY ioctl error: -22 md: import disk2: [8,32] (sdc) WDC WD10EACS-00D6B0 WD-WCAU40627966 offset: 63 size: 976762552 mdcmd (3): start md: import disk0: [8,48] (sdd) WDC WD10EACS-00ZJB0 WD-WCASJ1908386 offset: 63 size: 976762552 md: import disk1: HDIO_GET_IDENTITY ioctl error: -22 md: import disk2: [8,32] (sdc) WDC WD10EACS-00D6B0 WD-WCAU40627966 offset: 63 size: 976762552 unraid: allocated 7288kB md2: running, size: 976762552 blocks mdcmd (5): check md: recovery thread woken up ... md: recovery thread syncing parity disk ... # mkdir -p /mnt/foo # mount /dev/sdb1 /mnt/foo # df -H /mnt/foo Filesystem Size Used Avail Use% Mounted on /dev/sdb1 1.1T 646G 355G 65% /mnt/foo # ls -l /mnt/foo/media/recordings/TheSimpsons-* -rwx------ 2 root root 3492790148 Feb 3 21:00 TheSimpsons-IMarriedMarge-4536516-0.ts -rwx------ 2 root root 3513946112 Nov 22 21:00 TheSimpsons-JazzyandthePussycats-3128200-0.mpg -rwx------ 2 root root 3508084736 Nov 20 01:30 TheSimpsons-LifeontheFastLane-3083514-0.mpg -rwx------ 2 root root 3489337344 Nov 19 21:00 TheSimpsons-That90sShow-3083515-0.mpg #

Gigabyte GA-MA78G-DS3H Mainboard AM2+ 6 port SATA QTY 1 $62.99 = $63 newegg.com Syba SD-SA2PEX-2IR PCIe 2 port SATA QTY 5 $14.50 = $87 iwin2win.com AMD LE-1640 Athlon 64 45W 2.6GHz 1MB L2 QTY 1 $42.00 = $42 mwave.com Generic Dual 1GB DDR2 2 sticks 1GB memory QTY 1 $40.00 = $40 yourFavoriteStore $232 before shipping gets you 16 PCIe sata ports. Seems like an okay deal. I'm sure some have done better, but I wanted to share. Of course, make sure you pick a suitable enclosure with proper airflow and most importantly a power supply that has sufficient 12V current for disk spin-up (typically 1.75A - 2.5A per drive - check the drive specs). I happen to have an attraction to the CM Stacker 810 cases that I can't seem to get over, even though they have become quite expensive. The mainboard has built-in video, so get the 2GB so that the shared video memory does not eat into the 1GB range that Unraid uses. I'm not sure if it was necessary, but I did flash the Syba boards with the latest Non-Raid version of the BIOS. Some posts in this forum had suggested it. THIS SETUP WILL NOT WORK WITH A DUAL PROCESSOR - it will reboot on Unraid 4.3.3. See my other post titled " Unraid 4.3.3 kernel + Athlon 64 x2 + Sil3132 = REBOOT on write", if you must, for the details... http://lime-technology.com/forum/index.php?topic=2310.msg17736#msg17736 HTH

Perhaps you use "rsync" instead of "mv" to transfer the write cache to the "array proper"? -> rsync allows you to limit the bandwidth you are using for the writes with the "bwlimit" switch. -> Any large files that were interrupted by an unraid reboot would continue from the point they were interrupted. -> You wouldn't need crontab and your performance wouldn't crash when you're having trouble sleeping at 3:40 AM. -> Sometimes mv "loses" files if the right combinations of error conditions arise. I've been using this method for a write cache disk for almost a year now and it works well. The write cache disk is on each host (not the unraid array) and it writes to the unraid array using rsync across the network and a bwlimit=2048k. rsync also works fine with localhost and has good performance. My unraid array installs the rsync package and runs the daemon when it starts. There are a seemingly infinite number of switches on rsync to control how the file is transferred so that it can be done most efficiently. I'm not sure if it will work for everyone or not, but having lived with the "write problem" and worked around it - I wanted to share what has worked for me. This forum has done so much to help me, that I hope this post will be helpful in some way.