2 Disk failure, constant errors still occuring, can no longer access via smb other performance issues.

leon

By leon
in General Support

Recommended Posts

leon Noob

leon

Posted
Posted

28 drive data array

dual parity

 

One parity drive failure

One data drive failure

 

Both drives little over 2.5 Years old

Snippet of the on-going log.

 

Apr 14 15:20:39 vortex emhttpd: error: get_filesystem_status, 7329: Structure needs cleaning (117): scandir Structure needs cleaning
Apr 14 15:20:39 vortex kernel: XFS (dm-4): metadata I/O error in "xfs_imap_to_bp+0x50/0x70 [xfs]" at daddr 0x2f9dc740 len 32 error 117
Apr 14 15:20:40 vortex kernel: XFS (dm-4): Metadata corruption detected at xfs_buf_ioend+0xa9/0x384 [xfs], xfs_inode block 0x2f9dc740 xfs_inode_buf_verify
Apr 14 15:20:40 vortex kernel: XFS (dm-4): Unmount and run xfs_repair
Apr 14 15:20:40 vortex kernel: XFS (dm-4): First 128 bytes of corrupted metadata buffer:
Apr 14 15:20:40 vortex kernel: 00000000: 70 36 48 19 a8 5d 53 78 b6 e0 82 42 38 0a 87 d6  p6H..]Sx...B8...
Apr 14 15:20:40 vortex kernel: 00000010: f1 e4 a6 7c db 05 eb cd 8a 44 c2 fb 08 6a e7 ca  ...|.....D...j..
Apr 14 15:20:40 vortex kernel: 00000020: aa ed 92 c2 25 ff 9f 27 5e 70 47 c7 d1 b0 a7 94  ....%..'^pG.....
Apr 14 15:20:40 vortex kernel: 00000030: e9 0a 77 e9 58 65 d9 0f e0 c0 0f 6a e9 cf a9 65  ..w.Xe.....j...e
Apr 14 15:20:40 vortex kernel: 00000040: 96 36 06 18 a6 8d 4b 6b 52 e4 88 4a 4b de f7 9b  .6....KkR..JK...
Apr 14 15:20:40 vortex kernel: 00000050: b7 7e c1 55 2a 39 ba 1d 9f 56 13 2c ec 04 d9 32  .~.U*9...V.,...2
Apr 14 15:20:40 vortex kernel: 00000060: 38 d1 de 63 61 85 34 1c d8 77 af 1b da fe 91 3c  8..ca.4..w.....<
Apr 14 15:20:40 vortex kernel: 00000070: da c9 f7 73 ae 21 76 d1 e6 b8 69 36 ed 45 4a cc  ...s.!v...i6.EJ.
Apr 14 15:20:40 vortex kernel: XFS (dm-4): metadata I/O error in "xfs_imap_to_bp+0x50/0x70 [xfs]" at daddr 0x2f9dc740 len 32 error 117
Apr 14 15:20:40 vortex emhttpd: error: get_filesystem_status, 7329: Structure needs cleaning (117): scandir Structure needs cleaning
 

 

diag attached

vortex-diagnostics-20240414-1518.zip

Link to comment
leon Noob

leon

Posted
  • Author
Posted
5 hours ago, JorgeB said:

Check filesystem for disk5, run it without -n

Phase 1 - find and verify superblock... - reporting progress in intervals of 15 minutes
Phase 2 - using internal log - zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this.

Link to comment
leon Noob

leon

Posted
  • Author
Posted
1 hour ago, JorgeB said:

Use -L 

Phase 1 - find and verify superblock...
        - reporting progress in intervals of 15 minutes
Phase 2 - using internal log
        - zero log...
ALERT: The filesystem has valuable metadata changes in a log which is being
destroyed because the -L option was used.
        - 11:00:25: zeroing log - 238465 of 238465 blocks done
        - scan filesystem freespace and inode maps...
clearing needsrepair flag and regenerating metadata
sb_fdblocks 228253041, counted 232638101
        - 11:00:26: scanning filesystem freespace - 36 of 36 allocation groups done
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - 11:00:26: scanning agi unlinked lists - 36 of 36 allocation groups done
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 30
        - agno = 15
        - agno = 31
        - agno = 16
        - agno = 32
        - agno = 33
        - agno = 34
        - agno = 35
        - agno = 17
        - agno = 18
        - agno = 19
        - agno = 20
        - agno = 21
        - agno = 22
        - agno = 23
        - agno = 24
        - agno = 25
Metadata corruption detected at 0x438a03, xfs_inode block 0x2f9dc740/0x4000
Metadata corruption detected at 0x438a03, xfs_inode block 0x2f9dc760/0x4000
Metadata corruption detected at 0x438a03, xfs_inode block 0x2f874a40/0x4000
Metadata corruption detected at 0x438a03, xfs_inode block 0x2f874a60/0x4000
        - agno = 26
        - agno = 27
        - agno = 28
        - agno = 29
bad CRC for inode 797395520
bad magic number 0xc07 on inode 797395520
bad version number 0x50 on inode 797395520
bad next_unlinked 0xc9001a04 on inode 797395520
inode identifier 15486349304751396487 mismatch on inode 797395520
bad CRC for inode 797395521
bad magic number 0xfe87 on inode 797395521
bad version number 0x35 on inode 797395521
bad next_unlinked 0xc6b69fe5 on inode 797395521
inode identifier 6192169045301083795 mismatch on inode 797395521
bad CRC for inode 797395522
bad magic number 0x7193 on inode 797395522
bad version number 0x5b on inode 797395522
bad next_unlinked 0xc85fbf32 on inode 797395522
inode identifier 14844527802528255077 mismatch on inode 797395522
bad CRC for inode 797395523
bad magic number 0x60a on inode 797395523
bad version number 0xffffffed on inode 797395523
bad next_unlinked 0x63617398 on inode 797395523
inode identifier 2640384398663555207 mismatch on inode 797395523
bad CRC for inode 797395524
bad magic number 0x89db on inode 797395524
bad version number 0xffffffd5 on inode 797395524
bad next_unlinked 0xac043047 on inode 797395524
inode identifier 1522398218937641906 mismatch on inode 797395524
bad CRC for inode 797395525
bad magic number 0xdbf9 on inode 797395525
bad version number 0xffffffa5 on inode 797395525
bad next_unlinked 0x943e374f on inode 797395525
inode identifier 15852597528450964637 mismatch on inode 797395525
bad CRC for inode 797395526
bad magic number 0x4bd4 on inode 797395526
bad version number 0x59 on inode 797395526
bad next_unlinked 0xa5364caf on inode 797395526
inode identifier 10779674375943109514 mismatch on inode 797395526
bad CRC for inode 797395527
bad magic number 0x2f24 on inode 797395527
bad version number 0xffffffaf on inode 797395527
bad next_unlinked 0xde71109d on inode 797395527
inode identifier 4198015948291376069 mismatch on inode 797395527
bad CRC for inode 797395528
bad magic number 0x358e on inode 797395528
bad version number 0xffffffa5 on inode 797395528
bad next_unlinked 0x7972eb8a on inode 797395528
inode identifier 3275820270847548257 mismatch on inode 797395528
bad CRC for inode 797395529
bad magic number 0x8c95 on inode 797395529
bad version number 0xffffffdb on inode 797395529
bad next_unlinked 0xc1eed014 on inode 797395529
inode identifier 7263128412841935538 mismatch on inode 797395529
bad CRC for inode 797395530
bad magic number 0xa8d6 on inode 797395530
bad version number 0xffffffb5 on inode 797395530
bad next_unlinked 0x927e5075 on inode 797395530
inode identifier 11910271123323779578 mismatch on inode 797395530
bad CRC for inode 797395531
bad magic number 0x79f3 on inode 797395531
bad version number 0x5b on inode 797395531
bad next_unlinked 0x87e2ef0e on inode 797395531
inode identifier 13520072780868709617 mismatch on inode 797395531
bad CRC for inode 797395532
bad magic number 0xc5ea on inode 797395532
bad version number 0xffffffb5 on inode 797395532
inode identifier 7274616936606553602 mismatch on inode 797395532
bad CRC for inode 797395533
bad magic number 0x8894 on inode 797395533
bad version number 0xffffff82 on inode 797395533
bad next_unlinked 0x518bf771 on inode 797395533
inode identifier 8920035083261328851 mismatch on inode 797395533
bad CRC for inode 797395534
bad magic number 0xf47f on inode 797395534
bad version number 0x4 on inode 797395534
bad next_unlinked 0xd8729db6 on inode 797395534
inode identifier 5842202447604225828 mismatch on inode 797395534
bad CRC for inode 797395535
bad magic number 0x5f69 on inode 797395535
bad version number 0x11 on inode 797395535
bad next_unlinked 0xe88b1152 on inode 797395535
inode identifier 17199995376065117099 mismatch on inode 797395535
bad CRC for inode 797395536
bad magic number 0xdfae on inode 797395536
bad version number 0x17 on inode 797395536
inode identifier 6454512695759490733 mismatch on inode 797395536
bad CRC for inode 797395537
bad magic number 0xe957 on inode 797395537
bad version number 0x49 on inode 797395537
bad next_unlinked 0xf0515aad on inode 797395537
inode identifier 7337246436524601550 mismatch on inode 797395537
bad CRC for inode 797395538
bad magic number 0x7be9 on inode 797395538
bad version number 0xffffffe6 on inode 797395538
bad next_unlinked 0x89507bba on inode 797395538
inode identifier 2867619276478255675 mismatch on inode 797395538
bad CRC for inode 797395539
bad magic number 0xefa8 on inode 797395539
bad version number 0x44 on inode 797395539
bad next_unlinked 0xe0b9f460 on inode 797395539
inode identifier 2613973249383737038 mismatch on inode 797395539
bad CRC for inode 797395540
bad magic number 0xe987 on inode 797395540
bad version number 0xffffffef on inode 797395540
bad next_unlinked 0x5df13b9b on inode 797395540
inode identifier 11931639989252174670 mismatch on inode 797395540
bad CRC for inode 797395541
bad magic number 0x6c18 on inode 797395541
bad version number 0x3e on inode 797395541
bad next_unlinked 0xfbde2e95 on inode 797395541
inode identifier 16622613532722751831 mismatch on inode 797395541
bad CRC for inode 797395542
bad magic number 0x93f3 on inode 797395542
bad version number 0x64 on inode 797395542
bad next_unlinked 0x6aaa224b on inode 797395542
inode identifier 3309773464070948621 mismatch on inode 797395542
bad CRC for inode 797395543
bad magic number 0xd89d on inode 797395543
bad version number 0x38 on inode 797395543
inode identifier 10007832845103880964 mismatch on inode 797395543
bad CRC for inode 797395544
bad magic number 0xf542 on inode 797395544
bad version number 0xffffffa1 on inode 797395544
bad next_unlinked 0xd50edc09 on inode 797395544
inode identifier 12951084351893107474 mismatch on inode 797395544
bad CRC for inode 797395545
bad magic number 0x106c on inode 797395545
bad version number 0x65 on inode 797395545
bad next_unlinked 0xccd820ee on inode 797395545
inode identifier 10580047338450042676 mismatch on inode 797395545
bad CRC for inode 797395546
bad magic number 0x6f32 on inode 797395546
bad version number 0xffffff8e on inode 797395546
bad next_unlinked 0xf628aff5 on inode 797395546
inode identifier 8709782888052223739 mismatch on inode 797395546
bad CRC for inode 797395547
bad magic number 0xe156 on inode 797395547
bad version number 0x9 on inode 797395547
bad next_unlinked 0x819f65b0 on inode 797395547
inode identifier 14855289374989871770 mismatch on inode 797395547
bad CRC for inode 797395548
bad magic number 0xbbaf on inode 797395548
bad version number 0xffffffae on inode 797395548
bad next_unlinked 0x4e186d42 on inode 797395548
inode identifier 11755687109389947739 mismatch on inode 797395548
bad CRC for inode 797395549
bad magic number 0xfa44 on inode 797395549
bad version number 0x48 on inode 797395549
bad next_unlinked 0xceeeb837 on inode 797395549
inode identifier 12063390189428638182 mismatch on inode 797395549
bad CRC for inode 797395550
bad magic number 0x89fd on inode 797395550
bad version number 0xffffffbb on inode 797395550
bad next_unlinked 0x4304f121 on inode 797395550
inode identifier 3832201698570735615 mismatch on inode 797395550
bad CRC for inode 797395551
bad magic number 0x9740 on inode 797395551
bad version number 0x4 on inode 797395551
inode identifier 3884280201130310038 mismatch on inode 797395551
bad CRC for inode 797395552
bad magic number 0x9229 on inode 797395552
bad version number 0x4b on inode 797395552
bad next_unlinked 0x8d516f94 on inode 797395552
inode identifier 11249198770675568758 mismatch on inode 797395552
bad CRC for inode 797395553
bad magic number 0xb8f1 on inode 797395553
bad version number 0x7b on inode 797395553
bad next_unlinked 0xf24d8d6a on inode 797395553
inode identifier 5691489371187986073 mismatch on inode 797395553
bad CRC for inode 797395554
bad magic number 0x4137 on inode 797395554
bad version number 0xffffffb7 on inode 797395554
bad next_unlinked 0xae651f78 on inode 797395554
inode identifier 7741066826743996827 mismatch on inode 797395554
bad CRC for inode 797395555
bad magic number 0xaeb on inode 797395555
bad version number 0x49 on inode 797395555
bad next_unlinked 0xd87a7d23 on inode 797395555
inode identifier 2618287474903119965 mismatch on inode 797395555
bad CRC for inode 797395556
bad magic number 0xc8ea on inode 797395556
bad version number 0x48 on inode 797395556
bad next_unlinked 0xaf761c32 on inode 797395556
inode identifier 14336170147589198288 mismatch on inode 797395556
bad CRC for inode 797395557
bad magic number 0x8c0f on inode 797395557
bad version number 0xfffffffe on inode 797395557
inode identifier 10456145794095237034 mismatch on inode 797395557
bad CRC for inode 797395558
bad magic number 0xcd53 on inode 797395558
bad version number 0xffffffda on inode 797395558
bad next_unlinked 0x4cf0bd75 on inode 797395558
inode identifier 11417652496800043346 mismatch on inode 797395558
bad CRC for inode 797395559
bad magic number 0x8d82 on inode 797395559
bad version number 0xffffffd5 on inode 797395559
bad next_unlinked 0x3afe0306 on inode 797395559
inode identifier 17561548204093417445 mismatch on inode 797395559
bad CRC for inode 797395560
bad magic number 0x16bb on inode 797395560
bad version number 0x6a on inode 797395560
bad next_unlinked 0x66e6b880 on inode 797395560
inode identifier 7042557379019584013 mismatch on inode 797395560
bad CRC for inode 797395561
bad magic number 0x8ccd on inode 797395561
bad version number 0xfffffffc on inode 797395561
bad next_unlinked 0x6f9651b8 on inode 797395561
inode identifier 7022029629959570746 mismatch on inode 797395561
bad CRC for inode 797395562
bad magic number 0xce71 on inode 797395562
bad version number 0xffffff81 on inode 797395562
bad next_unlinked 0x9e4d3c9b on inode 797395562
inode identifier 14874002874533417198 mismatch on inode 797395562
bad CRC for inode 797395563
bad magic number 0xa3b2 on inode 797395563
bad version number 0x4d on inode 797395563
bad next_unlinked 0x7a799f33 on inode 797395563
inode identifier 16952025866311515896 mismatch on inode 797395563
bad CRC for inode 797395564
bad magic number 0x2971 on inode 797395564
bad version number 0xffffffb7 on inode 797395564
bad next_unlinked 0xd9906fff on inode 797395564
inode identifier 12712010939497171538 mismatch on inode 797395564
bad CRC for inode 797395565
bad magic number 0x276e on inode 797395565
bad version number 0xffffff85 on inode 797395565
bad next_unlinked 0xa3da36e6 on inode 797395565
inode identifier 12340854786383319604 mismatch on inode 797395565
bad CRC for inode 797395566
bad magic number 0x5875 on inode 797395566
bad version number 0xffffffa3 on inode 797395566
bad next_unlinked 0xf2316299 on inode 797395566
inode identifier 6623889657727105997 mismatch on inode 797395566
bad CRC for inode 797395567
bad magic number 0xd3d1 on inode 797395567
bad version number 0xffffffd2 on inode 797395567
bad next_unlinked 0x8c5117f5 on inode 797395567
inode identifier 9015457311458011640 mismatch on inode 797395567
bad CRC for inode 797395568
bad magic number 0x325f on inode 797395568
bad version number 0x4e on inode 797395568
bad next_unlinked 0x3e096083 on inode 797395568
inode identifier 17762859335895626971 mismatch on inode 797395568
bad CRC for inode 797395569
bad magic number 0x1491 on inode 797395569
bad version number 0x27 on inode 797395569
bad next_unlinked 0xd214006c on inode 797395569
inode identifier 18008101715645743351 mismatch on inode 797395569
bad CRC for inode 797395570
bad magic number 0x8887 on inode 797395570
bad version number 0x55 on inode 797395570
bad next_unlinked 0xdbd55a1f on inode 797395570
inode identifier 6054926225777569297 mismatch on inode 797395570
bad CRC for inode 797395571
bad magic number 0x66e7 on inode 797395571
bad version number 0x3d on inode 797395571
bad next_unlinked 0xb20e5720 on inode 797395571
inode identifier 10598225932513747546 mismatch on inode 797395571
bad CRC for inode 797395572
bad magic number 0x114b on inode 797395572
bad version number 0x3e on inode 797395572
inode identifier 1477277398721194808 mismatch on inode 797395572
bad CRC for inode 797395573
bad magic number 0xc989 on inode 797395573
bad version number 0xfffffff9 on inode 797395573
bad next_unlinked 0xa8a92673 on inode 797395573
inode identifier 5001243155945708493 mismatch on inode 797395573
bad CRC for inode 797395574
bad magic number 0x6d38 on inode 797395574
bad version number 0x7a on inode 797395574
inode identifier 15699063653440465667 mismatch on inode 797395574
bad CRC for inode 797395575
bad magic number 0xc193 on inode 797395575
bad version number 0x15 on inode 797395575
bad next_unlinked 0xa2885ae9 on inode 797395575
inode identifier 1734125816195745974 mismatch on inode 797395575
bad CRC for inode 797395576
bad magic number 0x9222 on inode 797395576
bad version number 0xffffffaa on inode 797395576
bad next_unlinked 0xe5d3afe6 on inode 797395576
inode identifier 10838843052037893165 mismatch on inode 797395576
bad CRC for inode 797395577
bad magic number 0x9297 on inode 797395577
bad version number 0xffffffa9 on inode 797395577
inode identifier 3878205069538778832 mismatch on inode 797395577
bad CRC for inode 797395578
bad magic number 0x796f on inode 797395578
bad version number 0xffffff98 on inode 797395578
bad next_unlinked 0x5967f370 on inode 797395578
inode identifier 9940037190909924350 mismatch on inode 797395578
bad CRC for inode 797395579
bad magic number 0x8d42 on inode 797395579
bad version number 0x3d on inode 797395579
bad next_unlinked 0x7dd5891e on inode 797395579
inode identifier 1147837267907261208 mismatch on inode 797395579
bad CRC for inode 797395580
bad magic number 0xe6c on inode 797395580
bad version number 0xffffffa2 on inode 797395580
bad next_unlinked 0xc8f8c878 on inode 797395580
inode identifier 2136673029997110885 mismatch on inode 797395580
bad CRC for inode 797395581
bad magic number 0xe10 on inode 797395581
bad version number 0xffffff8c on inode 797395581
bad next_unlinked 0xe655a91c on inode 797395581
inode identifier 8566334680483765438 mismatch on inode 797395581
bad CRC for inode 797395582
bad magic number 0x21e on inode 797395582
bad version number 0xffffffe7 on inode 797395582
bad next_unlinked 0x3ce48b4a on inode 797395582
inode identifier 16470854412033246515 mismatch on inode 797395582
bad CRC for inode 797395583
bad magic number 0x692c on inode 797395583
bad version number 0x12 on inode 797395583
inode identifier 16014660606841586275 mismatch on inode 797395583
bad CRC for inode 798869312
bad magic number 0x7036 on inode 798869312
bad version number 0xffffffa8 on inode 798869312
inode identifier 3109621660945003857 mismatch on inode 798869312
bad CRC for inode 798869313
bad magic number 0x5dd on inode 798869313
bad version number 0xffffff92 on inode 798869313
bad next_unlinked 0x584bad14 on inode 798869313
inode identifier 10053703877263303514 mismatch on inode 798869313
bad CRC for inode 798869314
bad magic number 0x75b7 on inode 798869314
bad version number 0xe on inode 798869314
bad next_unlinked 0x6f799f64 on inode 798869314
inode identifier 10806051536216527765 mismatch on inode 798869314
bad CRC for inode 798869315
bad magic number 0x461d on inode 798869315
bad version number 0x32 on inode 798869315
bad next_unlinked 0xd84128cf on inode 798869315
inode identifier 16005563008510368775 mismatch on inode 798869315
bad CRC for inode 798869316
bad magic number 0x6674 on inode 798869316
bad version number 0xffffff80 on inode 798869316
inode identifier 15806492834548361905 mismatch on inode 798869316
bad CRC for inode 798869317
bad magic number 0xcb13 on inode 798869317
bad version number 0xfffffff6 on inode 798869317
bad next_unlinked 0xecd5ac3d on inode 798869317
inode identifier 8172363539279604064 mismatch on inode 798869317
bad CRC for inode 798869318
bad magic number 0x2491 on inode 798869318
bad version number 0x4f on inode 798869318
inode identifier 11523380878475704297 mismatch on inode 798869318
bad CRC for inode 798869319
bad magic number 0x1cd2 on inode 798869319
bad version number 0x50 on inode 798869319
bad next_unlinked 0xe729ac67 on inode 798869319
inode identifier 16897671819279594412 mismatch on inode 798869319
bad CRC for inode 798869320
bad magic number 0xde7a on inode 798869320
bad version number 0xffffffa3 on inode 798869320
bad next_unlinked 0xb840133a on inode 798869320
inode identifier 16715952999008369690 mismatch on inode 798869320
bad CRC for inode 798869321
bad magic number 0xbb35 on inode 798869321
bad version number 0x41 on inode 798869321
bad next_unlinked 0x778d5890 on inode 798869321
inode identifier 7020638936436302241 mismatch on inode 798869321
bad CRC for inode 798869322
bad magic number 0x6411 on inode 798869322
bad version number 0xffffffb6 on inode 798869322
bad next_unlinked 0xe744122c on inode 798869322
inode identifier 4573735337418990653 mismatch on inode 798869322
bad CRC for inode 798869323
bad magic number 0xa928 on inode 798869323
bad version number 0xffffff91 on inode 798869323
bad next_unlinked 0xd10465a5 on inode 798869323
inode identifier 671309052995664691 mismatch on inode 798869323
bad CRC for inode 798869324
bad magic number 0x37fc on inode 798869324
bad version number 0xf on inode 798869324
bad next_unlinked 0xb211bb29 on inode 798869324
inode identifier 458893976505888031 mismatch on inode 798869324
bad CRC for inode 798869325
bad magic number 0xad84 on inode 798869325
bad version number 0xffffff84 on inode 798869325
inode identifier 3293728361104981591 mismatch on inode 798869325
bad CRC for inode 798869326
bad magic number 0x8ce4 on inode 798869326
bad version number 0xffffffd7 on inode 798869326
bad next_unlinked 0x72348fce on inode 798869326
inode identifier 16547954506037750716 mismatch on inode 798869326
bad CRC for inode 798869327
bad magic number 0x3967 on inode 798869327
bad version number 0x47 on inode 798869327
bad next_unlinked 0x757ef796 on inode 798869327
inode identifier 5426946970315592324 mismatch on inode 798869327
bad CRC for inode 798869328
bad magic number 0x7327 on inode 798869328
bad version number 0xffffff8c on inode 798869328
bad next_unlinked 0xc0a54843 on inode 798869328
inode identifier 11582770037805368884 mismatch on inode 798869328
bad CRC for inode 798869329
bad magic number 0xa7a0 on inode 798869329
bad version number 0x33 on inode 798869329
bad next_unlinked 0xdf9c32f1 on inode 798869329
inode identifier 13144306709341923824 mismatch on inode 798869329
bad CRC for inode 798869330
bad magic number 0x4591 on inode 798869330
bad version number 0x1f on inode 798869330
inode identifier 13730960878100428962 mismatch on inode 798869330
bad CRC for inode 798869331
bad magic number 0xfd28 on inode 798869331
bad version number 0x0 on inode 798869331
bad next_unlinked 0xc893c6d7 on inode 798869331
inode identifier 8653792320520055487 mismatch on inode 798869331
bad CRC for inode 798869332
bad magic number 0x844a on inode 798869332
bad version number 0xffffffdf on inode 798869332
bad next_unlinked 0xa4e55f52 on inode 798869332
inode identifier 498577407243183009 mismatch on inode 798869332
bad CRC for inode 798869333
bad magic number 0x4ce4 on inode 798869333
bad version number 0x64 on inode 798869333
bad next_unlinked 0x48633b27 on inode 798869333
inode identifier 12018296184730777233 mismatch on inode 798869333
bad CRC for inode 798869334
bad magic number 0x2252 on inode 798869334
bad version number 0xffffffe7 on inode 798869334
bad next_unlinked 0x4398eb96 on inode 798869334
inode identifier 11303215398000632080 mismatch on inode 798869334
bad CRC for inode 798869335
bad magic number 0xb401 on inode 798869335
bad version number 0x1a on inode 798869335
bad next_unlinked 0xfb853de1 on inode 798869335
inode identifier 12010047588593805100 mismatch on inode 798869335
bad CRC for inode 798869336
bad magic number 0x88bc on inode 798869336
bad version number 0xffffff96 on inode 798869336
bad next_unlinked 0x5941ebae on inode 798869336
inode identifier 3095149720311567248 mismatch on inode 798869336
bad CRC for inode 798869337
bad magic number 0x2b95 on inode 798869337
bad version number 0x9 on inode 798869337
bad next_unlinked 0x3dbdd432 on inode 798869337
inode identifier 11378569744298597366 mismatch on inode 798869337
bad CRC for inode 798869338
bad magic number 0xcad1 on inode 798869338
bad version number 0xffffffe8 on inode 798869338
bad next_unlinked 0xf89287c5 on inode 798869338
inode identifier 13228017754746104463 mismatch on inode 798869338
bad CRC for inode 798869339
bad magic number 0xb18a on inode 798869339
bad version number 0x6d on inode 798869339
bad next_unlinked 0x6f227c44 on inode 798869339
inode identifier 17153097478890950618 mismatch on inode 798869339
bad CRC for inode 798869340
bad magic number 0xb342 on inode 798869340
bad version number 0x53 on inode 798869340
inode identifier 224796725780967307 mismatch on inode 798869340
bad CRC for inode 798869341
bad magic number 0x487a on inode 798869341
bad version number 0x43 on inode 798869341
bad next_unlinked 0x8b32360f on inode 798869341
inode identifier 6050159671033398465 mismatch on inode 798869341
bad CRC for inode 798869342
bad magic number 0xf917 on inode 798869342
bad version number 0x24 on inode 798869342
bad next_unlinked 0xc8ed450a on inode 798869342
inode identifier 17134707397989242565 mismatch on inode 798869342
bad CRC for inode 798869343
bad magic number 0x87fa on inode 798869343
bad version number 0xffffffc6 on inode 798869343
bad next_unlinked 0xc1fd6395 on inode 798869343
inode identifier 3851257797457439738 mismatch on inode 798869343
bad CRC for inode 798869344
bad magic number 0xed22 on inode 798869344
bad version number 0x70 on inode 798869344
bad next_unlinked 0xd55f6808 on inode 798869344
inode identifier 6802550264366792422 mismatch on inode 798869344
bad CRC for inode 798869345
bad magic number 0xef3 on inode 798869345
bad version number 0xffffffb0 on inode 798869345
bad next_unlinked 0xe71636e3 on inode 798869345
inode identifier 8621234130638216277 mismatch on inode 798869345
bad CRC for inode 798869346
bad magic number 0x43bf on inode 798869346
bad version number 0x4a on inode 798869346
bad next_unlinked 0x9e1ad8c3 on inode 798869346
inode identifier 14275334542455837621 mismatch on inode 798869346
bad CRC for inode 798869347
bad magic number 0x3251 on inode 798869347
bad version number 0xffffff80 on inode 798869347
bad next_unlinked 0x797012b3 on inode 798869347
inode identifier 12004719413943487581 mismatch on inode 798869347
bad CRC for inode 798869348
bad magic number 0x459b on inode 798869348
bad version number 0x56 on inode 798869348
inode identifier 12189824397363734787 mismatch on inode 798869348
bad CRC for inode 798869349
bad magic number 0xcb15 on inode 798869349
bad version number 0x13 on inode 798869349
bad next_unlinked 0x77ce4fcb on inode 798869349
inode identifier 11132801695331193076 mismatch on inode 798869349
bad CRC for inode 798869350
bad magic number 0x148 on inode 798869350
bad version number 0x13 on inode 798869350
bad next_unlinked 0xfe80af76 on inode 798869350
inode identifier 5978604259049889853 mismatch on inode 798869350
bad CRC for inode 798869351
bad magic number 0x4b1b on inode 798869351
bad version number 0x78 on inode 798869351
bad next_unlinked 0x42170b60 on inode 798869351
inode identifier 16835533022402996067 mismatch on inode 798869351
bad CRC for inode 798869352
bad magic number 0x4102 on inode 798869352
bad version number 0x7b on inode 798869352
inode identifier 9144266785437089953 mismatch on inode 798869352
bad CRC for inode 798869353
bad magic number 0x8761 on inode 798869353
bad version number 0xffffff8b on inode 798869353
bad next_unlinked 0xfaaaaa56 on inode 798869353
inode identifier 8199023066634619812 mismatch on inode 798869353
bad CRC for inode 798869354
bad magic number 0xc196 on inode 798869354
bad version number 0x12 on inode 798869354
bad next_unlinked 0x8d55a5cb on inode 798869354
inode identifier 6765527913550169138 mismatch on inode 798869354
bad CRC for inode 798869355
bad magic number 0xa0c2 on inode 798869355
bad version number 0xffffff89 on inode 798869355
bad next_unlinked 0xa2433a31 on inode 798869355
inode identifier 16895515823861210958 mismatch on inode 798869355
bad CRC for inode 798869356
bad magic number 0x5624 on inode 798869356
bad version number 0xffffffde on inode 798869356
bad next_unlinked 0x65057d5b on inode 798869356
inode identifier 12394284838153524946 mismatch on inode 798869356
bad CRC for inode 798869357
bad magic number 0x781 on inode 798869357
bad version number 0x26 on inode 798869357
bad next_unlinked 0x4e48a7d2 on inode 798869357
inode identifier 2277596443584815960 mismatch on inode 798869357
bad CRC for inode 798869358
bad magic number 0x10c5 on inode 798869358
bad version number 0xffffffb6 on inode 798869358
bad next_unlinked 0x4cd2fa0a on inode 798869358
inode identifier 16080138059926067020 mismatch on inode 798869358
bad CRC for inode 798869359
bad magic number 0x65f0 on inode 798869359
bad version number 0xffffffc1 on inode 798869359
bad next_unlinked 0xe17caef5 on inode 798869359
inode identifier 8439432710010054877 mismatch on inode 798869359
bad CRC for inode 798869360
bad magic number 0x6db4 on inode 798869360
bad version number 0xffffffee on inode 798869360
bad next_unlinked 0x7cd100b0 on inode 798869360
inode identifier 9883920304414756862 mismatch on inode 798869360
bad CRC for inode 798869361
bad magic number 0x7e94 on inode 798869361
bad version number 0x3c on inode 798869361
inode identifier 9559384396643940709 mismatch on inode 798869361
bad CRC for inode 798869362
bad magic number 0xdb86 on inode 798869362
bad version number 0xffffffa4 on inode 798869362
bad next_unlinked 0xc6f13c70 on inode 798869362
inode identifier 16038829822077000382 mismatch on inode 798869362
bad CRC for inode 798869363
bad magic number 0x8f7c on inode 798869363
bad version number 0xffffff97 on inode 798869363
bad next_unlinked 0xf3c46c2b on inode 798869363
inode identifier 2147118257905312866 mismatch on inode 798869363
bad CRC for inode 798869364
bad magic number 0xbe20 on inode 798869364
bad version number 0xffffff82 on inode 798869364
bad next_unlinked 0xea6f726d on inode 798869364
inode identifier 106665377874393495 mismatch on inode 798869364
bad CRC for inode 798869365
bad magic number 0x6450 on inode 798869365
bad version number 0x42 on inode 798869365
bad next_unlinked 0xc9589d1a on inode 798869365
inode identifier 12651835169529939204 mismatch on inode 798869365
bad CRC for inode 798869366
bad magic number 0x5550 on inode 798869366
bad version number 0xffffff9b on inode 798869366
bad next_unlinked 0xafc638fb on inode 798869366
inode identifier 10848891386808015283 mismatch on inode 798869366
bad CRC for inode 798869367
bad magic number 0xf6ec on inode 798869367
bad version number 0x19 on inode 798869367
bad next_unlinked 0xd6c75ad4 on inode 798869367
inode identifier 4714556353019500074 mismatch on inode 798869367
bad CRC for inode 798869368
bad magic number 0x1f08 on inode 798869368
bad version number 0xffffffce on inode 798869368
bad next_unlinked 0x3b34c000 on inode 798869368
inode identifier 11013657666062846542 mismatch on inode 798869368
bad CRC for inode 798869369
bad magic number 0xb725 on inode 798869369
bad version number 0x4 on inode 798869369
inode identifier 18039731665365337891 mismatch on inode 798869369
bad CRC for inode 798869370
bad magic number 0xd757 on inode 798869370
bad version number 0xffffffb2 on inode 798869370
bad next_unlinked 0xf4b31a39 on inode 798869370
inode identifier 6650516410589666202 mismatch on inode 798869370
bad CRC for inode 798869371
bad magic number 0x365b on inode 798869371
bad version number 0x6e on inode 798869371
bad next_unlinked 0x8c5bb4a9 on inode 798869371
inode identifier 8150083091283083106 mismatch on inode 798869371
bad CRC for inode 798869372
bad magic number 0x1275 on inode 798869372
bad version number 0xffffffad on inode 798869372
bad next_unlinked 0xe46b0b11 on inode 798869372
inode identifier 1692284750240927367 mismatch on inode 798869372
bad CRC for inode 798869373
bad magic number 0xd720 on inode 798869373
bad version number 0x27 on inode 798869373
bad next_unlinked 0xbe1608a1 on inode 798869373
inode identifier 8850197755502664770 mismatch on inode 798869373
bad CRC for inode 798869374
bad magic number 0x42e2 on inode 798869374
bad version number 0xffffffa4 on inode 798869374
bad next_unlinked 0x6339194c on inode 798869374
inode identifier 12558193683514033533 mismatch on inode 798869374
bad CRC for inode 798869375
bad magic number 0x9897 on inode 798869375
bad version number 0x60 on inode 798869375
bad next_unlinked 0xfa60dbbb on inode 798869375
inode identifier 6119862303767686978 mismatch on inode 798869375
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - 11:00:29: process known inodes and inode discovery - 27072 of 27136 inodes done
        - process newly discovered inodes...
14c122b08680: Badness in key lookup (length)
bp=(bno 0x2f874a40, len 16384 bytes) key=(bno 0x2f874a40, len 4096 bytes)
14c122b08680: Badness in key lookup (length)
bp=(bno 0x2f874a60, len 16384 bytes) key=(bno 0x2f874a60, len 4096 bytes)
14c122b08680: Badness in key lookup (length)
bp=(bno 0x2f9dc740, len 16384 bytes) key=(bno 0x2f9dc740, len 4096 bytes)
14c122b08680: Badness in key lookup (length)
bp=(bno 0x2f9dc760, len 16384 bytes) key=(bno 0x2f9dc760, len 4096 bytes)
        - 11:00:29: process newly discovered inodes - 72 of 36 allocation groups done
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - 11:00:29: setting up duplicate extent list - 36 of 36 allocation groups done
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 7
        - agno = 11
        - agno = 16
        - agno = 25
        - agno = 32
        - agno = 6
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 2
        - agno = 12
        - agno = 3
        - agno = 14
        - agno = 15
        - agno = 13
entry "2019.03.07" at block 0 offset 768 in directory inode 7516192933 references non-existent inode 798869317
	clearing inode number in entry at offset 768...
        - agno = 18
        - agno = 19
        - agno = 22
        - agno = 20
        - agno = 21
        - agno = 24
        - agno = 23
        - agno = 4
        - agno = 26
        - agno = 27
entry "2018.09.23" at block 0 offset 456 in directory inode 21474836631 references non-existent inode 798869340
	clearing inode number in entry at offset 456...
        - agno = 28
entry "Season 7" in shortform directory 4297064588 references non-existent inode 798869372
junking entry "Season 7" in directory inode 4297064588
        - agno = 5
        - agno = 30
        - agno = 29
        - agno = 34
        - agno = 35
entry "Home Video and Pictures" in shortform directory 128 references non-existent inode 798869318
junking entry "Home Video and Pictures" in directory inode 128
        - agno = 17
entry "EFI-" in shortform directory 3221225628 references non-existent inode 798869316
junking entry "EFI-" in directory inode 3221225628
entry "Season 3" in shortform directory 37580963991 references non-existent inode 798869326
junking entry "Season 3" in directory inode 37580963991
entry "parity.check.tuning" at block 0 offset 904 in directory inode 1073741998 references non-existent inode 798869321
	clearing inode number in entry at offset 904...
        - agno = 31
        - agno = 33
entry "Season 6" in shortform directory 27917287574 references non-existent inode 798869343
entry "Season 6" in shortform directory 35433480320 references non-existent inode 798869348
junking entry "Season 6" in directory inode 35433480320
entry "dynamix.system.stats" at block 0 offset 2552 in directory inode 1073741998 references non-existent inode 798869342
	clearing inode number in entry at offset 2552...
junking entry "Season 6" in directory inode 27917287574
entry "Season 2" in shortform directory 36507222148 references non-existent inode 798869345
junking entry "Season 2" in directory inode 36507222148
entry "Maze Runner The Death Cure (2018)" at block 0 offset 656 in directory inode 3221237573 references non-existent inode 797395578
	clearing inode number in entry at offset 656...
entry "Bridge of Spies (2015)" at block 0 offset 736 in directory inode 3221237573 references non-existent inode 797395580
	clearing inode number in entry at offset 736...
entry "The Rookie (2002)" at block 0 offset 896 in directory inode 3221237573 references non-existent inode 797395582
	clearing inode number in entry at offset 896...
entry "After Earth (2013)" at block 0 offset 1120 in directory inode 3221237573 references non-existent inode 798869313
	clearing inode number in entry at offset 1120...
entry "Ambulance (2022)" at block 0 offset 4008 in directory inode 3221237573 references non-existent inode 798869339
	clearing inode number in entry at offset 4008...
entry "Sitter, The (2011)" at block 1 offset 640 in directory inode 3221237573 references non-existent inode 798869366
	clearing inode number in entry at offset 640...
entry "Scorpion" at block 0 offset 1184 in directory inode 1078722658 references non-existent inode 798869364
	clearing inode number in entry at offset 1184...
entry "Godfather of Harlem" at block 0 offset 1808 in directory inode 1078722658 references non-existent inode 798869315
	clearing inode number in entry at offset 1808...
entry "Star Trek - Strange New Worlds" at block 0 offset 3088 in directory inode 1078722658 references non-existent inode 798869338
	clearing inode number in entry at offset 3088...
entry "The Sopranos" at block 0 offset 3864 in directory inode 1078722658 references non-existent inode 798869312
	clearing inode number in entry at offset 3864...
entry "Season 3" in shortform directory 1078722668 references non-existent inode 798869327
junking entry "Season 3" in directory inode 1078722668
entry "2003-03" in shortform directory 3647392849 references non-existent inode 798869319
junking entry "2003-03" in directory inode 3647392849
entry "Season 7" in shortform directory 1176503287 references non-existent inode 798869359
junking entry "Season 7" in directory inode 1176503287
entry ".." at block 0 offset 80 in directory inode 1762648895 references non-existent inode 798869366
        - 11:00:29: check for inodes claiming duplicate blocks - 27008 of 27136 inodes done
Phase 5 - rebuild AG headers and trees...
        - 11:00:31: rebuild AG headers and trees - 36 of 36 allocation groups done
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
bad hash table for directory inode 21474836631 (no data entry): rebuilding
rebuilding directory inode 21474836631
bad hash table for directory inode 1073741998 (no data entry): rebuilding
rebuilding directory inode 1073741998
bad hash table for directory inode 1078722658 (no data entry): rebuilding
rebuilding directory inode 1078722658
entry ".." in directory inode 1762648895 points to non-existent inode 798869366
bad hash table for directory inode 1762648895 (no data entry): rebuilding
rebuilding directory inode 1762648895
bad hash table for directory inode 3221237573 (no data entry): rebuilding
rebuilding directory inode 3221237573
bad hash table for directory inode 7516192933 (no data entry): rebuilding
rebuilding directory inode 7516192933
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
disconnected dir inode 1762648832, moving to lost+found
disconnected dir inode 1762648848, moving to lost+found
disconnected dir inode 1762648888, moving to lost+found
disconnected dir inode 1762648889, moving to lost+found
disconnected dir inode 1762648895, moving to lost+found
disconnected dir inode 2149391535, moving to lost+found
disconnected dir inode 3647392849, moving to lost+found
disconnected dir inode 9689329765, moving to lost+found
Phase 7 - verify and correct link counts...
resetting inode 128 nlinks from 9 to 8
resetting inode 3221225628 nlinks from 8 to 7
resetting inode 7516192933 nlinks from 40 to 39
resetting inode 1073741998 nlinks from 26 to 24
resetting inode 21474836631 nlinks from 35 to 34
resetting inode 37580963991 nlinks from 3 to 2
resetting inode 3221237573 nlinks from 131 to 125
resetting inode 4297064588 nlinks from 8 to 7
resetting inode 27917287574 nlinks from 4 to 3
resetting inode 133 nlinks from 2 to 10
resetting inode 36507222148 nlinks from 4 to 3
resetting inode 1078722658 nlinks from 143 to 139
resetting inode 35433480320 nlinks from 4 to 3
resetting inode 1078722668 nlinks from 5 to 4
resetting inode 3647392849 nlinks from 4 to 3
resetting inode 1176503287 nlinks from 6 to 5
        - 11:00:34: verify and correct link counts - 36 of 36 allocation groups done
Maximum metadata LSN (1:1569747) is ahead of log (1:2).
Format log to cycle 4.
done

 

Link to comment
leon Noob

leon

Posted
  • Author
Posted
7 minutes ago, JorgeB said:

Should be fixed, look for a lost+found folder.

 

users shares are back, I can ls dir's like normal now. One parity disk and this disk5 are still marked as disabled due to failure. Should I just be replacing them?

Link to comment
JorgeB Grand Master

JorgeB

Posted
Posted

SMART looks fine for both disks, and two disks getting disabled at the same time is usually not a disk problem, it would be a very big coincidence, if it happens again download the diagnostics before rebooting, if you are happy with disk5 contents you can rebuild on top, and also re-sync parity at the same time:

 

https://docs.unraid.net/unraid-os/manual/storage-management#rebuilding-a-drive-onto-itself

Link to comment
leon Noob

leon

Posted
  • Author
Posted

Something is still not right .. my remaining parity drive is going nuts with errors.

 

image.thumb.png.5455ebd43a368d69bda6fa879e10c05c.png

 

Apr 16 05:48:56 vortex kernel: md: disk29 read error, sector=2942605784
Apr 16 05:48:56 vortex kernel: md: disk29 read error, sector=709420528
Apr 16 05:48:56 vortex kernel: md: disk29 read error, sector=1953601776
Apr 16 05:48:56 vortex kernel: md: disk29 read error, sector=4240
Apr 16 05:48:56 vortex kernel: md: disk29 read error, sector=709420536
Apr 16 05:48:56 vortex kernel: md: disk29 read error, sector=22472755704
Apr 16 05:48:56 vortex kernel: md: disk29 read error, sector=4248
Apr 16 05:48:56 vortex kernel: md: disk29 read error, sector=709420544
Apr 16 05:48:56 vortex kernel: md: disk29 read error, sector=1953601776
Apr 16 05:48:56 vortex kernel: md: disk29 read error, sector=1952543040
Apr 16 05:48:56 vortex kernel: md: disk29 read error, sector=709419672
Apr 16 05:48:56 vortex kernel: md: disk29 read error, sector=22472755704

 

 

Can you tell of Both parities and desk 5 are on the same controller? trying to understand what is going on here.

Link to comment
leon Noob

leon

Posted
  • Author
Posted

DUAL PSU setup so seems unlikely

image.png.5085608153dec0931e9aeb4f188e8b0f.png

 

But I guess.. connection problem is possible though its in a 4u box with drives slotted in so no cables I've been messing with.., sigh backplane issue maybe

Link to comment
leon Noob

leon

Posted
  • Author
Posted

Yes, I swapped the disks around in the chassis and the known good disk are now failing in random ways.  Not marked as disabled.. but clearly.. broken 

root@vortex:/mnt# ls
/bin/ls: cannot access 'disk17': Input/output error
/bin/ls: cannot access 'disk3': Input/output error
addons/  disk10/  disk12/  disk14/  disk16/  disk18/  disk2/   disk21/  disk23/  disk25/  disk27/  disk4/  disk6/  disk8/  disks/    rootshare/  user/
disk1/   disk11/  disk13/  disk15/  disk17/  disk19/  disk20/  disk22/  disk24/  disk26/  disk3/   disk5/  disk7/  disk9/  remotes/  ssdcache/   user0/

 

Link to comment
  • 2 weeks later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing