October 13, 20232 yr I have 5 disks (14TB & 10TB) and dual parity (18TB). i7-3770K, 32GB RAM, on a UPS etc... Over the weekend I've had 1 disk get corrupted, then 2 disks get disabled. Timeline (roughly): 10/6/2023: Parity check running 10/7/2023: Presumably around here errors started happening as I have bunch of emails from AppdataBackup failing Description: Array has 3 disks with read errors Importance: warning Disk 1 - WDC_WD140EDGZ-11B1PA0_Y5KYWNNC (sdi) (errors 885744) Disk 3 - WDC_WD100EMAZ-00WJTA0_JEGW40YN (sdh) (errors 885744) Disk 5 - WDC_WD100EMAZ-00WJTA0_JEGW1ADN (sdg) (errors 2048) 10/8/2023 ~4AM (diagnostics attached before next reboot): * **disk1 (WDC_WD140EDGZ-11B1PA0_Y5KYWNNC) is disabled** * **disk5 (WDC_WD100EMAZ-00WJTA0_JEGW1ADN) is disabled** * **disk1 (WDC_WD140EDGZ-11B1PA0_Y5KYWNNC) has read errors** * **disk3 (WDC_WD100EMAZ-00WJTA0_JEGW40YN) has read errors** * **disk5 (WDC_WD100EMAZ-00WJTA0_JEGW1ADN) has read errors** * **/var/log is getting full (currently 100 % used)** * **Unable to write to disk1** 10/8/2023 9:30PM: * **disk1 (WDC_WD140EDGZ-11B1PA0_Y5KYWNNC) is disabled** * **disk5 (WDC_WD100EMAZ-00WJTA0_JEGW1ADN) is disabled** Disk 3 - WDC_WD100EMAZ-00WJTA0_JEGW40YN (sde) (errors 108928) 10/8/2023 10PM: I pull out disk1 and swap for another in case I mess something up Disk 1, is being reconstructed and is available for normal operation Data-Rebuild started 10/9/2023 12AM: Parity - ST18000NE000-2YY101_ZR54LMDX (sdc) - active 45 C [OK] Parity 2 - ST18000NM000J-2TV103_ZR56TDY7 (sdb) - active 45 C [OK] Disk 1 - WDC_WD140EDGZ-11B2DA2_2CHDSP1P (sdk) - active 34 C [DISK INVALID] Disk 2 - WDC_WD140EDGZ-11B1PA0_Y6G0EEUC (sdh) - active 37 C [OK] Disk 3 - WDC_WD100EMAZ-00WJTA0_JEGW40YN (sde) - active 33 C (disk has read errors) [NOK] Disk 4 - ST10000DM0004-2GR11L_ZJV6AXZB (sdd) - active 33 C (disk has read errors) [NOK] Disk 5 - WDC_WD100EMAZ-00WJTA0_JEGW1ADN (sdf) - active 30 C [DISK DSBL] Cache - Samsung_SSD_860_EVO_250GB_S59WNMFN703104R (sdi) - active 41 C [OK] Cache 2 - CT1000MX500SSD1_2307E6ABC2BB (sdj) - active 34 C [OK] Also note the DISK INVALID issue, seemingly need an XFS_REPAIR. I attempted that on previous disk before pulling it out and just trying to do parity rebuild. 10/12/2023 1:45AM Parity done: Disk 1 returned to normal operation Description: WDC_WD140EDGZ-11B2DA2_2CHDSP1P (sdk) Data-Rebuild finished (3299065 errors) Description: Duration: 3 days, 3 hours, 20 minutes, 25 seconds. Average speed: 51.6 MB/s <NOTE: STILL HAVE Unmountable: Unsupported or no file system for Disk 1> 10/12/2023 2AM start reconstruction on disk5: Disk 5, is being reconstructed and is available for normal operation Description: WDC_WD100EMAZ-00WJTA0_JEGW1ADN (sdf) Event: Unraid Data-Rebuild 10/13/2023 2AM: Disk 5 in error state (disk dsbl) Description: WDC_WD100EMAZ-00WJTA0_JEGW1ADN (sdf) - Disk 3 in error state (disk dsbl) Description: WDC_WD100EMAZ-00WJTA0_JEGW40YN (sde) Description: Array has 3 disks with read errors Disk 3 - WDC_WD100EMAZ-00WJTA0_JEGW40YN (sde) (errors 64) Disk 4 - ST10000DM0004-2GR11L_ZJV6AXZB (sdd) (errors 156) Disk 5 - WDC_WD100EMAZ-00WJTA0_JEGW1ADN (sdf) (errors 1024) 10/13/2023 ~11AM Diag attached: Data-Rebuild finished (3299065 errors) (obviously incorrect) Description: Duration: 3 days, 3 hours, 20 minutes, 25 seconds. Average speed: 51.6 MB/s Rebooted to change around cables. After reboot: Disk 1 - WDC_WD140EDGZ-11B2DA2_2CHDSP1P (sdj) (errors * **disk3 (WDC_WD100EMAZ-00WJTA0_JEGW40YN) is disabled** * **disk5 (WDC_WD100EMAZ-00WJTA0_JEGW1ADN) is disabled** 10/13/2023 12PM: I tried doing a recheck but the noises coming from disk1 and errors popping up make me think not best way forward. Disk 1 - WDC_WD140EDGZ-11B2DA2_2CHDSP1P (sdj) (errors 990668) Disk 4 - ST10000DM0004-2GR11L_ZJV6AXZB (sdf) (errors 607524) Currently: Disk 1: Officially available, but (1) Corrupted file system (2) making noises and log has errors Oct 13 11:48:29 kernel: I/O error, dev sdj, sector 84870416 op 0x0:(READ) flags 0x0 phys_seg 128 prio class 2 Oct 13 11:48:29 kernel: device offline error, dev sdj, sector 84871440 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 2 Oct 13 11:48:30 kernel: sd 11:0:4:0: [sdj] Synchronizing SCSI cache Oct 13 11:48:30 kernel: sd 11:0:4:0: [sdj] Synchronize Cache(10) failed: Result: hostbyte=0x01 driverbyte=DRIVER_OK Disk 3: Emulated & Disabled Disk 4: Had some Reallocated Sector accounts last week, but stopped at 136. Planned on replacing soon. Disk 5: Emulated & Disabled Me: Confused bridge-diagnostics-20231008-2035.zip bridge-diagnostics-20231013-1216.zip bridge-diagnostics-20231013-1034.zip
October 13, 20232 yr Community Expert You are having what look like power/connection issues with multiple disks in different controllers, suggest not doing any rebuilds/xfs_repair before fixing that, keep the old replaced disk intact if you still have it, after checking/replacing cables next step would be using a different PSU if possible,.
October 13, 20232 yr Author 2 hours ago, JorgeB said: You are having what look like power/connection issues with multiple disks in different controllers, suggest not doing any rebuilds/xfs_repair before fixing that, keep the old replaced disk intact if you still have it, after checking/replacing cables next step would be using a different PSU if possible,. So to confirm you'd recommend I should try diff PSU/cables and then rerun Read-Check? Edited October 13, 20232 yr by 128bytes
October 14, 20232 yr Community Expert 14 hours ago, 128bytes said: So to confirm you'd recommend I should try diff PSU/cables and then rerun Read-Check? Correct.
October 18, 20232 yr Author On 10/14/2023 at 4:42 AM, JorgeB said: Correct. So Read-Check is about to finish, 0 errors so far. Assuming that completes without errors, what's the next step?
October 18, 20232 yr Author 55 minutes ago, JorgeB said: Post new diags once it finishes, with the array started. Completed with no errors! (Guess the old power supply is garbage) UI hasn't changed, I still have: Disk 1 Unmountable DIsk 3 disabled Disk 5 disabled And my option by rebuilt parity is still just Read-Check. bridge-diagnostics-20231018-1147.zip Edited October 18, 20232 yr by 128bytes
October 18, 20232 yr Community Expert Check filesystem on disk1, run it without -n, and if it asks for it use -L To rebuild the disable the disabled disks, assuming the emulated disks content looks correct see here, you can do both at the same time: https://docs.unraid.net/unraid-os/manual/storage-management#rebuilding-a-drive-onto-itself
October 18, 20232 yr Author 51 minutes ago, JorgeB said: Check filesystem on disk1, run it without -n, and if it asks for it use -L 1) This is xfs_repair below, just wanted to make sure seems ok to you. 2) Parity main bottleneck is drives right? I plan on upgrading the CPU on this (3rd gen i7 3770k to 12th gen i5-12600K 6p4e), just wanted to know if I should bother doing the upgrade first. Phase 1 - find and verify superblock... - block cache size set to 1419048 entries Phase 2 - using internal log - zero log... Log inconsistent (didn't find previous header) failed to find log head zero_log: cannot find log head/tail (xlog_find_tail=5) - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 1 - agno = 4 - agno = 0 - agno = 3 - agno = 5 - agno = 6 - agno = 7 - agno = 2 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 Phase 5 - rebuild AG headers and trees... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (6:207268) is ahead of log (1:2). Format log to cycle 9. XFS_REPAIR Summary Wed Oct 18 12:23:00 2023 Phase Start End Duration Phase 1: 10/18 12:18:08 10/18 12:18:08 Phase 2: 10/18 12:18:08 10/18 12:19:10 1 minute, 2 seconds Phase 3: 10/18 12:19:10 10/18 12:19:18 8 seconds Phase 4: 10/18 12:19:18 10/18 12:19:18 Phase 5: 10/18 12:19:18 10/18 12:19:19 1 second Phase 6: 10/18 12:19:19 10/18 12:19:26 7 seconds Phase 7: 10/18 12:19:26 10/18 12:19:26 Total run time: 1 minute, 18 seconds done bridge-diagnostics-20231018-1302.zip Edited October 18, 20232 yr by 128bytes
October 18, 20232 yr Author 59 minutes ago, JorgeB said: https://docs.unraid.net/unraid-os/manual/storage-management#rebuilding-a-drive-onto-itself Also, bug report (maybe just bad wording?): I started in maintenance mode and did the unassign/assign and the notifications tell me its being reconstructed even though I didnt hit sync yet. Event: Unraid Disk 3 message Subject: Notice [BRIDGE] - Disk 3, is being reconstructed and is available for normal operation Description: WDC_WD100EMAZ-00WJTA0_JEGW40YN (sdd) Importance: normal Event: Unraid Disk 5 message Subject: Notice [BRIDGE] - Disk 5, is being reconstructed and is available for normal operation Description: WDC_WD100EMAZ-00WJTA0_JEGW1ADN (sde) Importance: normal bridge-diagnostics-20231018-1311.zip
October 18, 20232 yr Community Expert 20 minutes ago, 128bytes said: I started in maintenance mode and did the unassign/assign and the notifications tell me its being reconstructed even though I didnt hit sync yet. No need to use maintenance mode for that, and in normal mode the rebuild begins immedicably after array start.
October 18, 20232 yr Author 6 minutes ago, JorgeB said: No need to use maintenance mode for that, and in normal mode the rebuild begins immedicably after array start. Yes I did maintenance just to keep it offline for now. I'm saying the message sent though implies the rebuild started.
October 18, 20232 yr Community Expert 5 minutes ago, 128bytes said: I'm saying the message sent though implies the rebuild started. I assume it's expecting the server to be started in normal mode, it can be considered a bug, but it's a corner case and really not important.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.