September 9, 20169 yr Hey all, I was in the process of copying some files over to one of my shares and all of a sudden I got an error that I didn't have permission. I am on Windows 10 so I hit cancel and tried again. I was prompted to enter my username and password which kept getting denied. Upon checking the share page on unRAID GUI, all my shares are missing now. This is the log: Sep 8 21:54:40 Tower kernel: BTRFS info (device loop0): no csum found for inode 14072 start 3686400 Sep 8 21:54:40 Tower kernel: BTRFS info (device loop0): no csum found for inode 14072 start 3690496 Sep 8 21:54:40 Tower kernel: BTRFS info (device loop0): no csum found for inode 14072 start 3694592 Sep 8 21:54:40 Tower kernel: BTRFS info (device loop0): no csum found for inode 14072 start 3698688 Sep 8 21:54:40 Tower kernel: BTRFS info (device loop0): no csum found for inode 14072 start 3702784 Sep 8 21:54:40 Tower kernel: BTRFS info (device loop0): no csum found for inode 14072 start 3706880 Sep 8 21:54:40 Tower kernel: BTRFS info (device loop0): no csum found for inode 14072 start 1617920 Sep 8 21:55:02 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 21:55:32 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 21:56:02 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 21:56:32 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 21:57:02 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 21:57:32 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 21:58:02 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 21:58:32 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 21:59:02 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 21:59:32 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 22:00:02 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 22:00:33 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 22:01:03 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:27 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:28 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:28 Tower shfs/user: shfs_write: write: (5) Input/output error Sep 8 22:01:33 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 22:02:03 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 22:02:33 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 22:02:49 Tower emhttp: get_fs_sizes: /mnt/user/VM statfs: Input/output error Sep 8 22:02:49 Tower emhttp: get_fs_sizes: /mnt/user/domains statfs: Input/output error Sep 8 22:02:49 Tower emhttp: get_fs_sizes: /mnt/user/system statfs: Input/output error Sep 8 22:02:54 Tower emhttp: get_fs_sizes: /mnt/user/VM statfs: Input/output error Sep 8 22:02:54 Tower emhttp: get_fs_sizes: /mnt/user/domains statfs: Input/output error Sep 8 22:02:54 Tower emhttp: get_fs_sizes: /mnt/user/system statfs: Input/output error Sep 8 22:02:54 Tower kernel: blk_update_request: 3780 callbacks suppressed Sep 8 22:02:54 Tower kernel: blk_update_request: I/O error, dev loop0, sector 1961792 Sep 8 22:02:54 Tower kernel: btrfs_dev_stat_print_on_error: 3714 callbacks suppressed Sep 8 22:02:54 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 2, rd 5703, flush 0, corrupt 0, gen 0 Sep 8 22:02:54 Tower kernel: blk_update_request: I/O error, dev loop0, sector 4058944 Sep 8 22:02:54 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 2, rd 5704, flush 0, corrupt 0, gen 0 Sep 8 22:02:54 Tower kernel: blk_update_request: I/O error, dev loop0, sector 1961792 Sep 8 22:02:54 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 2, rd 5705, flush 0, corrupt 0, gen 0 Sep 8 22:02:54 Tower kernel: blk_update_request: I/O error, dev loop0, sector 4058944 Sep 8 22:02:54 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 2, rd 5706, flush 0, corrupt 0, gen 0 Sep 8 22:02:54 Tower kernel: blk_update_request: I/O error, dev loop0, sector 1961792 Sep 8 22:02:54 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 2, rd 5707, flush 0, corrupt 0, gen 0 Sep 8 22:02:54 Tower kernel: blk_update_request: I/O error, dev loop0, sector 4058944 Sep 8 22:02:54 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 2, rd 5708, flush 0, corrupt 0, gen 0 Sep 8 22:03:03 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 22:03:12 Tower emhttp: get_fs_sizes: /mnt/user/VM statfs: Input/output error Sep 8 22:03:12 Tower emhttp: get_fs_sizes: /mnt/user/domains statfs: Input/output error Sep 8 22:03:12 Tower emhttp: get_fs_sizes: /mnt/user/system statfs: Input/output error Sep 8 22:03:33 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 22:04:03 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 22:04:33 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 22:05:03 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 22:05:16 Tower emhttp: get_fs_sizes: /mnt/user/VM statfs: Input/output error Sep 8 22:05:16 Tower emhttp: get_fs_sizes: /mnt/user/domains statfs: Input/output error Sep 8 22:05:16 Tower emhttp: get_fs_sizes: /mnt/user/system statfs: Input/output error Sep 8 22:05:20 Tower emhttp: get_fs_sizes: /mnt/user/VM statfs: Input/output error Sep 8 22:05:20 Tower emhttp: get_fs_sizes: /mnt/user/domains statfs: Input/output error Sep 8 22:05:20 Tower emhttp: get_fs_sizes: /mnt/user/system statfs: Input/output error Sep 8 22:05:33 Tower emhttp: get_fs_sizes: /mnt/user/VM statfs: Input/output error Sep 8 22:05:33 Tower emhttp: get_fs_sizes: /mnt/user/domains statfs: Input/output error Sep 8 22:05:33 Tower emhttp: get_fs_sizes: /mnt/user/system statfs: Input/output error Sep 8 22:05:33 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 22:05:45 Tower emhttp: get_fs_sizes: /mnt/user/VM statfs: Input/output error Sep 8 22:05:45 Tower emhttp: get_fs_sizes: /mnt/user/domains statfs: Input/output error Sep 8 22:05:45 Tower emhttp: get_fs_sizes: /mnt/user/system statfs: Input/output error Sep 8 22:05:45 Tower kernel: blk_update_request: I/O error, dev loop0, sector 1961792 Sep 8 22:05:45 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 2, rd 5709, flush 0, corrupt 0, gen 0 Sep 8 22:05:45 Tower kernel: blk_update_request: I/O error, dev loop0, sector 4058944 Sep 8 22:05:45 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 2, rd 5710, flush 0, corrupt 0, gen 0 Sep 8 22:05:45 Tower kernel: blk_update_request: I/O error, dev loop0, sector 1961792 Sep 8 22:05:45 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 2, rd 5711, flush 0, corrupt 0, gen 0 Sep 8 22:05:45 Tower kernel: blk_update_request: I/O error, dev loop0, sector 4058944 Sep 8 22:05:45 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 2, rd 5712, flush 0, corrupt 0, gen 0 Sep 8 22:05:55 Tower emhttp: get_fs_sizes: /mnt/user/VM statfs: Input/output error Sep 8 22:05:55 Tower emhttp: get_fs_sizes: /mnt/user/domains statfs: Input/output error Sep 8 22:05:55 Tower emhttp: get_fs_sizes: /mnt/user/system statfs: Input/output error Sep 8 22:06:01 Tower emhttp: get_fs_sizes: /mnt/user/VM statfs: Input/output error Sep 8 22:06:01 Tower emhttp: get_fs_sizes: /mnt/user/domains statfs: Input/output error Sep 8 22:06:01 Tower emhttp: get_fs_sizes: /mnt/user/system statfs: Input/output error Sep 8 22:06:03 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 22:06:09 Tower emhttp: get_fs_sizes: /mnt/user/VM statfs: Input/output error Sep 8 22:06:09 Tower emhttp: get_fs_sizes: /mnt/user/domains statfs: Input/output error Sep 8 22:06:09 Tower emhttp: get_fs_sizes: /mnt/user/system statfs: Input/output error Sep 8 22:06:11 Tower emhttp: get_fs_sizes: /mnt/user/VM statfs: Input/output error Sep 8 22:06:11 Tower emhttp: get_fs_sizes: /mnt/user/domains statfs: Input/output error Sep 8 22:06:11 Tower emhttp: get_fs_sizes: /mnt/user/system statfs: Input/output error Sep 8 22:06:13 Tower emhttp: get_fs_sizes: /mnt/user/VM statfs: Input/output error Sep 8 22:06:13 Tower emhttp: get_fs_sizes: /mnt/user/domains statfs: Input/output error Sep 8 22:06:13 Tower emhttp: get_fs_sizes: /mnt/user/system statfs: Input/output error Sep 8 22:06:17 Tower emhttp: get_fs_sizes: /mnt/user/VM statfs: Input/output error Sep 8 22:06:17 Tower emhttp: get_fs_sizes: /mnt/user/domains statfs: Input/output error Sep 8 22:06:17 Tower emhttp: get_fs_sizes: /mnt/user/system statfs: Input/output error Sep 8 22:06:33 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 22:07:04 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 22:07:34 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 22:07:58 Tower emhttp: get_fs_sizes: /mnt/user/VM statfs: Input/output error Sep 8 22:07:58 Tower emhttp: get_fs_sizes: /mnt/user/domains statfs: Input/output error Sep 8 22:07:58 Tower emhttp: get_fs_sizes: /mnt/user/system statfs: Input/output error Sep 8 22:08:02 Tower emhttp: get_fs_sizes: /mnt/user/VM statfs: Input/output error Sep 8 22:08:02 Tower emhttp: get_fs_sizes: /mnt/user/domains statfs: Input/output error Sep 8 22:08:02 Tower emhttp: get_fs_sizes: /mnt/user/system statfs: Input/output error Sep 8 22:08:04 Tower kernel: XFS (md1): xfs_log_force: error -5 returned. Sep 8 22:08:28 Tower emhttp: get_fs_sizes: /mnt/user/VM statfs: Input/output error Sep 8 22:08:28 Tower emhttp: get_fs_sizes: /mnt/user/domains statfs: Input/output error Sep 8 22:08:28 Tower emhttp: get_fs_sizes: /mnt/user/system statfs: Input/output error Can someone please help me figure out what is going on here? VM, appdata, domains, and system are the only things showing in the user shares section now. tower-diagnostics-20160908-2208.zip
September 9, 20169 yr Author I brought the array down and back up again and now disk 1 is not mountable. I am guessing the drive has failed me. Is that what this log is telling me?
September 9, 20169 yr Community Expert I brought the array down and back up again and now disk 1 is not mountable. I am guessing the drive has failed me. Is that what this log is telling me? a disk not being mountable rarely means it has physically failed. Far more typical is that a write failed that resulted in some level of file system level corruption. Since you initial post suggests a disk unexpectedly dropped offline this is certainly a possibility. Assuming the disk can be seen at the physical level after a reboot this can normally recovered by running the appropriate recovery tool for the file system type used on the drive. If you provide your system diagnostics (Tools->diagnostics) then that will provide information that will mean more informed guidance can be given on the actual status of the disk in question and thus the best next steps.
September 9, 20169 yr Author Thanks for the quick reply. I did attach my diagnostics in the first post. If you're not able to see the file or something I can upload it to Google drive to share.
September 9, 20169 yr Community Expert Your parity disk needs to be replaced. Next you'll need to run xfs_repair on disk1. Your cache disk also needs replacing.
September 9, 20169 yr Author Is the offline sector count the indicator of the parity disk failing? What would cause the file system to go corrupt in the middle of a transfer? I knew the WD Green drives probably weren't the best for this application but I didn't expect anything like this to happen. Are the two issues related or just coincidence? Thanks for the help though. Disk 2 doesn't have any files on ot. Is it possible to swap that in as the parity drive?
September 9, 20169 yr Community Expert Pending sectors > 0 = replace disk Device Model: WDC WD20EARS-00MVWB0 Serial Number: WD-WCAZAC767743 ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 21 3 Spin_Up_Time 0x0027 192 165 021 Pre-fail Always - 5391 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 80 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 1 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 056 056 000 Old_age Always - 32835 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 73 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 64 193 Load_Cycle_Count 0x0032 001 001 000 Old_age Always - 3292457 194 Temperature_Celsius 0x0022 118 099 000 Old_age Always - 32 196 Reallocated_Event_Count 0x0032 199 199 000 Old_age Always - 1 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 9 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 9 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 195 195 000 Old_age Offline - 1525 Model Family: Seagate Barracuda 7200.10 Device Model: ST3160815AS Serial Number: 9RX4FYAZ ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 118 093 006 Pre-fail Always - 199519550 3 Spin_Up_Time 0x0003 099 097 070 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 225 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 9 7 Seek_Error_Rate 0x000f 087 060 030 Pre-fail Always - 610326538 9 Power_On_Hours 0x0032 066 066 000 Old_age Always - 30047 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 224 187 Reported_Uncorrect 0x0032 001 001 000 Old_age Always - 1174 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 064 035 045 Old_age Always In_the_past 36 (Min/Max 28/37) 194 Temperature_Celsius 0x0022 036 065 000 Old_age Always - 36 (0 17 0 0 0) 195 Hardware_ECC_Recovered 0x001a 065 052 000 Old_age Always - 162145724 197 Current_Pending_Sector 0x0012 097 096 000 Old_age Always - 74 198 Offline_Uncorrectable 0x0010 097 096 000 Old_age Offline - 74 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 8 200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0 202 Data_Address_Mark_Errs 0x0032 100 253 000 Old_age Always - 0 You can use disk2 as parity, you need to do a new config.
September 9, 20169 yr Your parity disk needs to be replaced. Next you'll need to run xfs_repair on disk1. Your cache disk also needs replacing. The docker.img file is trashed, but that'll get fixed once you replace the cache disk. Additionally, I think that all of your plugins are way behind on updates (especially CA). You need to update CA, as your version probably won't even work anymore. You really should enable autoupdates on some of the plugins, and set up notifications so that you won't wind up in this situation again with either the plugins or with the multiple disk failures.
September 9, 20169 yr Author Squid, I went through and updated all of the plug-ins last week when I upgraded to version 6.2. I'm not sure where you are seeing thay they aren't up to date, but the CA plug in was not working after the update so I did manually update that. What do you mean by the docker image is trashed? Basically I'm still trying to understand what and how it happened. I did notice the bad sector count, but how does that have an effect on what happened here? The docker.img isnt on disk 1, so how did that end up trashed in all of this?
September 9, 20169 yr Squid, I went through and updated all of the plug-ins last week when I upgraded to version 6.2. I'm not sure where u pu are seeing thay they aren't up to date, but the CA plug in was not working after the update so I did manually update that. What do you mean by the docker image is trashed? Basically I'm still trying to understand what and how it happened. I did notice the bad sector count, but how does that have an effect on what happened here? The docker.img isnt on disk 1, so how did that end up trashed in all of this? Sorry I missed that you updated. Was looking at what installed when you booted on the 1st. All those errors listed as Sep 8 22:05:45 Tower kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 2, rd 5709, flush 0, corrupt 0, gen 0 Sep 8 22:05:45 Tower kernel: blk_update_request: I/O error, dev loop0, sector 4058944 Are referring to the docker.img file (loop0) Your cache disk also needs replacing. You've got 74 pending sectors on the cache drive (which johnnie also pointed out), and which are sectors that the drive wants to reallocate as bad, but hasn't gotten around to it yet. Those could be the problem with the docker.img, or the image could simply be corrupted for some other reason (too small? 10Gig tends to be small for most uses) Either way, the docker.img is trashed and needs to be rebuilt. Super easy to do that and not lose anything, but you should worry about the parity drive and disk1 first.
September 9, 20169 yr Author Okay that makes more sense now. In the system log, I see disk0 read errors starting on Sept 4th. Is that the start of my problems? I guess what I am going to do is plus the parity disk and replace it with disk2 (disk2 has nothing on it). I will then rebuild the parity and run xfs_repair on disk1. After disk1 is repaired I will rerun a parity check and swap out the cache drive for another drive I have laying around, and make the docker.img larger. Does that seem like a solid plan or do I have to do this in a different order? Sep 3 04:40:01 Tower root: Community Applications Auto Update Running Sep 3 21:01:41 Tower rpc.mountd[2256]: authenticated mount request from 10.0.0.146:716 for /mnt/user/Media (/mnt/user/Media) Sep 4 03:09:47 Tower kernel: ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Sep 4 03:09:47 Tower kernel: ata4.00: irq_stat 0x40000001 Sep 4 03:09:47 Tower kernel: ata4.00: failed command: READ DMA EXT Sep 4 03:09:47 Tower kernel: ata4.00: cmd 25/00:40:00:d7:d0/00:05:c6:00:00/e0 tag 0 dma 688128 in Sep 4 03:09:47 Tower kernel: res 51/40:bf:78:d7:d0/00:04:c6:00:00/e0 Emask 0x9 (media error) Sep 4 03:09:47 Tower kernel: ata4.00: status: { DRDY ERR } Sep 4 03:09:47 Tower kernel: ata4.00: error: { UNC } Sep 4 03:09:47 Tower kernel: ata4.00: configured for UDMA/133 Sep 4 03:09:47 Tower kernel: sd 5:0:0:0: [sde] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 Sep 4 03:09:47 Tower kernel: sd 5:0:0:0: [sde] tag#0 Sense Key : 0x3 [current] [descriptor] Sep 4 03:09:47 Tower kernel: sd 5:0:0:0: [sde] tag#0 ASC=0x11 ASCQ=0x4 Sep 4 03:09:47 Tower kernel: sd 5:0:0:0: [sde] tag#0 CDB: opcode=0x28 28 00 c6 d0 d7 00 00 05 40 00 Sep 4 03:09:47 Tower kernel: blk_update_request: I/O error, dev sde, sector 3335575416 Sep 4 03:09:47 Tower kernel: ata4: EH complete Sep 4 03:09:47 Tower kernel: md: disk0 read error, sector=3335575352 Sep 4 03:09:47 Tower kernel: md: disk0 read error, sector=3335575360 Sep 4 03:09:47 Tower kernel: md: disk0 read error, sector=3335575368
September 9, 20169 yr I guess what I am going to do is plus the parity disk and replace it with disk2 (disk2 has nothing on it). I will then rebuild the parity and run xfs_repair on disk1. After disk1 is repaired I will rerun a parity check and swap out the cache drive for another drive I have laying around, and make the docker.img larger. Does that seem like a solid plan or do I have to do this in a different order? Seems fine to me. You will have to do a new config to accomplish this, which means that if another drive drops dead in the process you will have lost data. But johnnie is the real expert around here on this kind of stuff.
September 9, 20169 yr Author Well thanks for taking a look at this and giving me some help here. I am on eBay looking for some replacement drives right now and I think I am going to just buy some enterprise class drives from WD. The RE4 drives look like a good balance between MTBF and price, as the 2TB models are going for ~$50 each. At 2M hours, that's 228 years MTBF so hopefully I can find some that haven't been abused too bad. I just bought these WD Green drives six months ago. I do have one more question though. What is the disk0 that the log if referring to in my post at 4:35? Sep 4 03:09:47 Tower kernel: md: disk0 read error, sector=3335575352 Sep 4 03:09:47 Tower kernel: md: disk0 read error, sector=3335575360 Sep 4 03:09:47 Tower kernel: md: disk0 read error, sector=3335575368
September 9, 20169 yr Well thanks for taking a look at this and giving me some help here. I am on eBay looking for some replacement drives right now and I think I am going to just buy some enterprise class drives from WD. The RE4 drives look like a good balance between MTBF and price, as the 2TB models are going for ~$50 each. At 2M hours, that's 228 years MTBF so hopefully I can find some that haven't been abused too bad. I just bought these WD Green drives six months ago. I do have one more question though. What is the disk0 that the log if referring to in my post at 4:35? Sep 4 03:09:47 Tower kernel: md: disk0 read error, sector=3335575352 Sep 4 03:09:47 Tower kernel: md: disk0 read error, sector=3335575360 Sep 4 03:09:47 Tower kernel: md: disk0 read error, sector=3335575368 Parity Drive And take MTBF with a grain of salt, because its a statistic, and like all statistics is actually meaningless (at least without knowing the sample sizes). Here's the MTBF of a person: What is the MTBF of an 25 year old human being? 70 years? 80? No, it’s actually over 800 years which highlights the difference between lifetime and MTBF. Take a large population of, say, 500,000 over a year, and seeing how many ‘failed’ (died) that year – e.g. 600 – so the failure rate is 600 per 500,000 ‘people-years’, i.e. 0.12% per year and the MTBF is the inverse of that which is 830 years. An individual won’t last that long, they will wear out long before then (unless they are Doctor Who), but for the population as a whole, in that ‘high reliability’ portion of their lifespan, it holds true – in a typical year you will only have to ‘replace’ 600 of them.
September 9, 20169 yr Author Thanks for the info. I see you run a wide variety of brands and everyone is going to have a different opinion on what is and isn't good to buy. Usually I am skeptical of used drives but money is tight right now and I have always had luck with WD. The Seagate cache drive that is failing me now is over 10 years old, and I never hear anyone saying good things about Seagate. I don't want to take this thread off on another direction, but if you know of a better drive in this price/capacity range, please let me know or shoot me a pm. I am worried more about longevity for these RAID drives and will probably throw in a 750GB Raptor that I have for my cache drive. Again, thanks for the help here. I don't know much about xfs and the inner workings of unRAID as I only seem to make time to learn when there is a problem, but I am going to spend whatever free time I have at work today learning up on this. I just hope all my data is intact when this is all said and done.
September 9, 20169 yr Community Expert Make sure you enable system notifications, you would have received a warning when the first read errors happened.
September 9, 20169 yr and I never hear anyone saying good things about Seagate. lol I'm the lone wolf out here. I only ever buy Seagates as I have zero issues with them. The other drives in my sig tend to be from years ago. (except for a single WD 3tb that I couldn't refuse the price on)
September 9, 20169 yr Author squid, I have never had an issue with Seagates either. I always thought they offered a good balance of price/performance myself. johnnie, I will enable notifications as soon as I get back up and running. I need to run xfs_repair from command line, correct? This isn't going to affect any data on the drive will it?
September 9, 20169 yr Community Expert You can use the WebGUI, though I prefer using the command line. This isn't going to affect any data on the drive will it? Running xfs_repair in read mode won't affect any data, when attempting a repair depends on how badly corrupt the file system is, there always a chance of data loss.
September 9, 20169 yr Author Johnnie, why do you suggest replacing the parity drive first? My thinking is that I should fix disk1 filesystem first, then replace the cache disk and allow unRAID to rebuild the drive from parity, then replace the parity drive (disk0) and run the parity checker finally. This will eliminate the worry of losing data from disk1. Is the parity drive too far gone at this point? Doesn't SMART reallocate the bad sectors? I understand this drive is no longer reliable but can't I still use it to rebuild the cache and disk1 if needed? squid, I am sure you have heard of Backblaze before. Check out their findings on Seagate, it seems pretty consistent year to year. Regardless, I have quite a few 250GB-500GB laptop and desktop drives laying around that all still work. This is the first Seagate that has failed me through the years, and I have three dead WD. I've always been reluctant to go to Hitachi. https://www.backblaze.com/blog/best-hard-drive-q4-2014/
September 9, 20169 yr Community Expert Johnnie, why do you suggest replacing the parity drive first? My thinking is that I should fix disk1 filesystem first, then replace the cache disk and allow unRAID to rebuild the drive from parity, then replace the parity drive (disk0) and run the parity checker finally. This will eliminate the worry of losing data from disk1. Is the parity drive too far gone at this point? Doesn't SMART reallocate the bad sectors? I understand this drive is no longer reliable but can't I still use it to rebuild the cache and disk1 if needed? Order doesn't matter, either way works, but unassign your current parity first, you shouldn't run xfx_repair with a bad parity disk.
September 9, 20169 yr squid, I am sure you have heard of Backblaze before. Check out their findings on Seagate, it seems pretty consistent year to year. Regardless, I have quite a few 250GB-500GB laptop and desktop drives laying around that all still work. This is the first Seagate that has failed me through the years, and I have three dead WD. I've always been reluctant to go to Hitachi. https://www.backblaze.com/blog/best-hard-drive-q4-2014/ Well aware of backblaze and their findings. Trouble with backblaze stats is that they admittedly use the drives outside of their specifications (ie: is a Corvette less reliable than a Jeep simply because a Corvette won't last 5 seconds off-road? I don't think so), and they never classify what a "failure" is which makes the stat unreliable. Big problem with using consumer grade hard drives in a real RAID environment is because of the operation of TLER. Because of that firmware change between enterprise and consumer drives, a consumer drive is indeed going to drop off of the RAID far quicker (and be classified as a failure) than an enterprise drive. This operation of drives doesn't affect unRaid because it is not a true RAID solution. IIRC Backblaze has now begun to switch to using NAS drives which is beginning to level out the playing field simply because they are more suited to a true RAID. (And in unRaid, short of the generally longer warranty there is no difference as the mechanics are the same, but the firmware has changed between regular drives and NAS drives because it is not a true RAID solution) Seagate's main problem is that they were the first to begin to introduce a NAS type drive with their 7200.11 series (1tb and 1.5tb) drives. And there was a mistake in the firmware which caused the drives to fail. Simple firmware update fixed that problem, but the legacy of a ton of people experiencing failures still lingers. In short, my experiences with hard drives tells me that WD is terrible. Yours (and others) experiences may differ. But I expect that I will continue to be a lone wolf and sing Seagate praise while trashing WD whenever I can while others shake their head at me. And I'm ok with that
September 9, 20169 yr Author Johnnie, why do you suggest replacing the parity drive first? My thinking is that I should fix disk1 filesystem first, then replace the cache disk and allow unRAID to rebuild the drive from parity, then replace the parity drive (disk0) and run the parity checker finally. This will eliminate the worry of losing data from disk1. Is the parity drive too far gone at this point? Doesn't SMART reallocate the bad sectors? I understand this drive is no longer reliable but can't I still use it to rebuild the cache and disk1 if needed? Order doesn't matter, either way works, but unassign your current parity first, you shouldn't run xfx_repair with a bad parity disk. Can you elaborate on why a little? Just trying to understand the mechanics of this all. squid, that is true and like everything else, ymmv. I didn't realize unRAID is unaffected by the NAS specific firmware in those types of drives. So basically, there is no reason to go with a NAS specific hard drive (ie WD Red, Seagate NAS) over say a WD black (not sure of Seagates performance series)?
September 9, 20169 yr Community Expert Can you elaborate on why a little? Just trying to understand the mechanics of this all. Parity will be updated with any modifications made by xfs_repair on disk1, since the disk is known to be bad, don't use it your it could compromise the file system recovery.
September 9, 20169 yr Johnnie, why do you suggest replacing the parity drive first? My thinking is that I should fix disk1 filesystem first, then replace the cache disk and allow unRAID to rebuild the drive from parity, then replace the parity drive (disk0) and run the parity checker finally. This will eliminate the worry of losing data from disk1. Is the parity drive too far gone at this point? Doesn't SMART reallocate the bad sectors? I understand this drive is no longer reliable but can't I still use it to rebuild the cache and disk1 if needed? Order doesn't matter, either way works, but unassign your current parity first, you shouldn't run xfx_repair with a bad parity disk. Can you elaborate on why a little? Just trying to understand the mechanics of this all. squid, that is true and like everything else, ymmv. I didn't realize unRAID is unaffected by the NAS specific firmware in those types of drives. So basically, there is no reason to go with a NAS specific hard drive (ie WD Red, Seagate NAS) over say a WD black (not sure of Seagates performance series)? The biggest change in NAS drives is the TLER which AFAIK only affects raid controllers (which unRaid doesn't support) https://en.wikipedia.org/wiki/Error_recovery_control but they do also have a longer warranty. My attitude is that any and all hard drives are going to fail. With the amount of hard drives I've got running in the server's and other computers, I expect one to die every year (warranty or not). When that doesn't happen (and it actually hasn't happened in 3 years now) I consider it a bonus. Personally, I don't consider a reallocated sector to be a failure. Growing # of reallocated, sure. But not one or two. But then again, I'm old enough to remember the days of ST-506 when drives had the bad sectors marked on the top of them from the factory, and you wouldn't trust a drive that didn't have at least one reallocated. Like you said, your mileage is going to vary, and people's opinions will vary.
Archived
This topic is now archived and is closed to further replies.