November 25, 20169 yr Last night, I had a functioning unRaid 2.6.4 array of 3 devices, one each as Storage (2TB), Parity (2TB), and Cache (500GB). This morning, I opened the webGUI to check on some downloads I had queued on a Deluge docker. CheckCommonErrors gave me a red notice indicating that a disc was either full, or mounted as read-only. Which is weird, the server was fine last night, and webGUI reports the array still has ~700GB free. So I check Deluge to find ~70 errors. Nothing is comming down, only a few files are being uploaded. So I telnet over to the server to check the discs from a command line. I find the following: /bin/ls: cannot access 'user0': Input/output error /bin/ls: cannot access 'disk1': Input/output error Ok, time to try something different. I have 2 X 2TB drives sitting in Unassigned Drives. They have both been pre-cleared and are ready for use. Recalling the initial message of my disc being full I imagine now is the time to add these disks to the array. So I stopped the array, grew it from 3 to 5 devices, and added the two precleared discs to Disc2 and Disc3. When I started the array, it showed Disc1, Disc2, and Disc3 as all 'Unmountable'. And at that point, I decided my fixes are not making things better, so I shut down the server. I hadn't noticed any 'red balls' at any stage, and am confused as to what is even happening. Of course it wasn't until after shut-down that I thought to start taking screenshots. Attached are the screenshots of the telnet session, and of the present assigned drives of my array. Please help me get this server back up.
November 25, 20169 yr Community Expert When I started the array, it showed Disc1, Disc2, and Disc3 as all 'Unmountable'. That is expected for disks 2 and 3, they need to be formatted, disk1 probably has filesystem corruption. Post the diagnostics: Tools -> Diagnostics
November 25, 20169 yr Author New Diags, now with useful data? unfocused-diagnostics-20161125-1303.zip
November 25, 20169 yr Community Expert Yes, disk1 filesystem needs to be checked, disks 2 and 3 need to be formatted, but don't formatted them now your disk1 will also be formatted, so: re-start array in maitnance mode and run xfs_repair: xfs_repair -v /dev/md1 After that if disk1 mounts you can formatted the other disks.
November 25, 20169 yr Author xfs_repair aborted with the following error: root@unFOCUSED:~# xfs_repair -v /dev/md1 Phase 1 - find and verify superblock... - block cache size set to 722032 entries Phase 2 - using internal log - zero log... zero_log: head block 1454017 tail block 1454008 ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this.
November 25, 20169 yr Community Expert Since it can't be mounted you need to use -L, usually there's no data loss.
November 25, 20169 yr Author Repair Completed, Disks 2 and 3 formatted. Array up without loss of important data... Deluge seems to have lost all of its working data, but that is replaceable. Thanks! root@unFOCUSED:~# xfs_repair -vL /dev/md1 Phase 1 - find and verify superblock... - block cache size set to 722032 entries Phase 2 - using internal log - zero log... zero_log: head block 1454017 tail block 1454008 ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... Metadata corruption detected at xfs_agf block 0x74704441/0x200 flfirst 118 in agf 2 too large (max = 118) sb_icount 12480, counted 12992 sb_ifree 123, counted 189 sb_fdblocks 290762031, counted 285542517 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 1 - agno = 2 - agno = 3 - agno = 0 Phase 5 - rebuild AG headers and trees... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (2:1454001) is ahead of log (1:2). Format log to cycle 5. XFS_REPAIR Summary Fri Nov 25 13:34:23 2016 Phase Start End Duration Phase 1: 11/25 13:33:11 11/25 13:33:11 Phase 2: 11/25 13:33:11 11/25 13:33:29 18 seconds Phase 3: 11/25 13:33:29 11/25 13:33:43 14 seconds Phase 4: 11/25 13:33:43 11/25 13:33:43 Phase 5: 11/25 13:33:43 11/25 13:33:43 Phase 6: 11/25 13:33:43 11/25 13:33:45 2 seconds Phase 7: 11/25 13:33:45 11/25 13:33:45 Total run time: 34 seconds done
Archived
This topic is now archived and is closed to further replies.