November 8, 201213 yr Hi everybody, a disk of my array got red ballled. i checked the smart report and it is healthy and I am pretty certain it is a cable problem. Since I was writing to the disk at the time, I though it would be safer if I rebuilt it and that is what I am doing now (I am at 2%). Should I worry that the disk in saying unformatted? It also shows as DISK_INVALID in unmenu and unmounted. Thanks in advance
November 8, 201213 yr Author Hi again, just finished rebuilding. All drives have a green ball and the array is running but the drive in question is still showing unformatted and is not mounted. Doing a file system check I get: Sorry, no file system detected on /dev/md9 Have I lost the data? I have a spare HD of th same size. Should I replace the problematic drive and rebuild or there is no point to it now? Any help would be appreciated.... syslog-2012-11-8.txt
November 8, 201213 yr Don't do anything until you analyze why disk9 will not mount. (unRAID labels ANY drive it cannot mount as "unformatted" regardless of the actual reason) Can you run reiserfsck --check on /dev/md9 You'll probably need to follow the steps in the wiki before you can run it. http://lime-technology.com/wiki/index.php/Check_Disk_Filesystems It might be that the file system is corrupted. If so, you can fix it and then it will be able to be mounted. It might be that the physical disk was not properly logically-connected to /dev/md9 If so, it is a different issue. I doubt this, but it is possible the MBR on the replacement disk is pointing to the wrong partition start. We've seen this issue on 4.7 (it is one of the outstanding bugs) If it complains of a missing superblock, DO NOT use reiserfsck to rebuild it, not yet at least... We want to make sure it is looking for it in the right place. To determine, you need to run a few commands on /dev/sdj (the physical disk connected as disk9) fdisk -lu /dev/sdj and dd if=/dev/sdj count=195 | od -c -A d | sed 30q You are looking for a line of output that contains a string that looks like this: 0097840 ^ \0 001 \0 R e I s E r 2 F s \0 \0 \0 or 0098352 ^ \0 001 \0 R e I s E r 2 F s \0 \0 \0 the reiserfs string is usually at address 0097840 (start of sector 63) or 0098352 (start of sector 64) but might be different on 3TB and larger disks that do not use an MBR partition. If you do not see an "R e I s E r 2 F s" string in the output, try these alternate commands: dd if=/dev/sdj count=195 | od -c -A d | sed '/R\ \ \ e\ \ \ I\ \ \ s\ \ \ E\ \ \ r\ \ \ 2\ \ \ F\ \ \ s/q' (There are three spaces between each letter in the string. The backslashes are optional. You could probably type: dd if=/dev/sdj count=195 | od -c -A d | sed '/R e I s E r 2 F s/q' As long as you type R space space space e space space space I space space space s ...... etc... Easiest might just be to cut and paste the command. If the MBR points to the wrong start sector, the disk will not mount. Your commands will help in learning if it points correctly. If not pointing to the correct starting sector, it is really easy to fix and not lose all your data. If it points to the correct starting sector and the superblock just needs to be fixed, that too is not hard to correct in most cases. Joe L.
November 8, 201213 yr Author Thanks for your reply. May I add that just before writing this, we had a mini-blackout and when unraid came back it started a parity check. I stopped it promptly but it obviously ran a bit. Hope it does not make things worse (although the moment i stopped it it was reporting 0 errors). So, here we go: 1. Before I ran reiserfsck I followed the instructions in the wiki but I could not umount /dev/md9 : umount: /dev/md9: not found I guess it couldn't be mounted since it seemed unformatted. I then ran reiserfsck --check /dev/md9 and, naturally: Failed to open the device '/dev/md9': No such file or directory I also tried reiserfsck --check /dev/sdj: reiserfs_open: the reiserfs superblock cannot be found on /dev/sdj. Failed to open the filesystem. 2. I then ran fdisk -lu /dev/sdj: Disk /dev/sdj: 2000.3 GB, 2000398934016 bytes 1 heads, 63 sectors/track, 62016336 cylinders, total 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdj1 63 3907029167 1953514552+ 83 Linux Partition 1 does not end on cylinder boundary. and dd if=/dev/sdj count=195 | od -c -A d | sed 30q root@Tower:~# dd if=/dev/sdj count=195 | od -c -A d | sed 30q 0000000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0000448 \0 \0 203 \0 \0 \0 ? \0 \0 \0 q 210 340 350 \0 \0 0000464 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0000496 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 U 252 0000512 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0097792 , 332 c 221 E 016 226 ? 335 e 2 m \n 230 355 W 0097808 311 c E 207 \n 370 U Y 206 374 275 260 D 342 t u 0097824 377 F + 327 b S 223 276 035 334 265 201 Y ^ } 3 0097840 307 332 366 361 364 2 214 > w 234 ] 037 [ X i p 0097856 m 205 Q @ W 255 037 366 275 356 R 310 023 236 321 w 0097872 257 211 % 034 [ 246 214 026 177 + b 325 / 361 242 ' 0097888 c 372 ) 354 \t 257 355 @ Y . s 343 B 310 # v 0097904 331 030 235 304 2 341 274 1 277 320 304 8 ` 254 X 020 0097920 265 a @ * 307 274 u 267 004 004 021 M \b 252 u 214 0097936 s p < 215 345 220 E , 023 4 s 024 002 \b c $ 0097952 b 252 y & 252 306 H 365 025 026 031 005 246 217 ^ \a 0097968 371 | 314 254 c 020 240 021 327 201 s 221 017 Z 303 1 0097984 021 300 354 y \ . ! N n 343 R 021 ] 376 234 . 0098000 027 V n 343 030 004 ] 376 i @ X 333 s P 263 \v 0098016 347 305 372 243 } 267 212 H 5 C 201 301 025 Z \t 0098032 217 304 T 035 351 & U ! 324 021 H 210 U 263 A 244 0098048 003 327 \t B 217 225 001 0 S 227 211 210 M 364 240 227 0098064 277 312 323 J F ` : 001 372 264 370 | 330 033 i 241 0098080 I 8 001 4 031 u 005 / 327 267 Q \t 220 > 325 } 0098096 231 264 232 222 275 F 204 4 250 256 D 225 d 034 002 020 0098112 352 B 343 ) q 215 225 210 315 261 002 025 027 \b 0 0098128 \ 212 310 \b 7 032 201 261 } 021 210 0 y 313 D 1 195+0 records in 195+0 records out 99840 bytes (100 kB) copied, 0.0040271 s, 24.8 MB/s I am afraid I did not see anything like what you expected. Finally I tried dd if=/dev/sdj count=195 | od -c -A d | sed '/R\ \ \ e\ \ \ I\ \ \ s\ \ \ E\ \ \ r\ \ \ 2\ \ \ F\ \ \ s/q' root@Tower:~# dd if=/dev/sdj count=195 | od -c -A d | sed '/R e I s E r 2 F s/q' 0000000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0000448 \0 \0 203 \0 \0 \0 ? \0 \0 \0 q 210 340 350 \0 \0 0000464 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0000496 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 U 252 0000512 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0097792 , 332 c 221 E 016 226 ? 335 e 2 m \n 230 355 W 0097808 311 c E 207 \n 370 U Y 206 374 275 260 D 342 t u 0097824 377 F + 327 b S 223 276 035 334 265 201 Y ^ } 3 0097840 307 332 366 361 364 2 214 > w 234 ] 037 [ X i p 0097856 m 205 Q @ W 255 037 366 275 356 R 310 023 236 321 w 0097872 257 211 % 034 [ 246 214 026 177 + b 325 / 361 242 ' 0097888 c 372 ) 354 \t 257 355 @ Y . s 343 B 310 # v 0097904 331 030 235 304 2 341 274 1 277 320 304 8 ` 254 X 020 0097920 265 a @ * 307 274 u 267 004 004 021 M \b 252 u 214 0097936 s p < 215 345 220 E , 023 4 s 024 002 \b c $ 0097952 b 252 y & 252 306 H 365 025 026 031 005 246 217 ^ \a 0097968 371 | 314 254 c 020 240 021 327 201 s 221 017 Z 303 1 0097984 021 300 354 y \ . ! N n 343 R 021 ] 376 234 . 0098000 027 V n 343 030 004 ] 376 i @ X 333 s P 263 \v 0098016 347 305 372 243 } 267 212 H 5 C 201 301 025 Z \t 0098032 217 304 T 035 351 & U ! 324 021 H 210 U 263 A 244 0098048 003 327 \t B 217 225 001 0 S 227 211 210 M 364 240 227 0098064 277 312 323 J F ` : 001 372 264 370 | 330 033 i 241 0098080 I 8 001 4 031 u 005 / 327 267 Q \t 220 > 325 } 0098096 231 264 232 222 275 F 204 4 250 256 D 225 d 034 002 020 0098112 352 B 343 ) q 215 225 210 315 261 002 025 027 \b 0 0098128 \ 212 310 \b 7 032 201 261 } 021 210 0 y 313 D 1 0098144 \b % 202 L k 347 254 232 335 331 235 3 275 \a 203 \0 0098160 274 375 016 242 m 020 n \0 375 q 341 226 w 210 016 342 0098176 032 E 252 ! 334 017 \b 212 x 0 f o \n \b ) 0098192 m 246 005 244 370 d h 311 t I 027 l 274 A ^ 0098208 @ L Q 245 213 d % > + 270 } 347 220 226 334 327 0098224 ` 353 333 $ 270 363 337 ] 371 z 277 u 350 272 270 9 0098240 s 216 . V I ( 356 330 ' m g 237 340 } 9 232 0098256 237 o 302 314 202 256 ` \0 257 355 304 \t v 0 224 265 0098272 016 376 376 241 ! 003 272 237 230 366 305 304 u 216 241 \f 0098288 e \a 005 004 326 030 361 6 364 324 340 332 \n , 375 210 0098304 243 032 " ! d 326 232 207 331 316 315 p 022 216 [ 202 0098320 310 307 347 203 312 256 , A 323 / / 201 200 030 r ! 0098336 346 306 020 204 l 355 ) 261 V M 374 x 032 032 304 234 0098352 357 036 246 227 \a \n ! 017 } v 300 206 213 ^ 274 264 0098368 D 230 210 1 370 \b ! 217 343 1 214 \0 203 347 361 0098384 t , \a 376 \v 201 206 3 234 006 D 221 306 F 352 $ 0098400 245 257 . F 004 n A 5 315 230 $ 212 330 ^ 261 210 0098416 265 276 R m 206 6 331 020 2 U 361 001 _ 321 o z 0098432 033 6 035 217 z Y 027 V 333 234 \ 264 u y 372 J 0098448 " 344 303 r 033 230 204 256 ? F A / 317 w 254 366 0098464 316 305 267 031 002 M L , 251 N 252 6 @ D 026 $ 0098480 177 x 324 254 x 006 244 320 P \v $ 224 r f @ I 0098496 351 327 \0 224 332 \f 021 B 311 6 I @ 332 \v G 365 0098512 201 \a [ X D 337 E & 342 212 242 213 W ! 027 317 0098528 224 275 022 _ z 353 \t ( @ 242 ? 023 * 025 \n b 0098544 f 025 \0 320 } 024 \0 ! B 6 021 247 215 031 k 312 0098560 303 006 " 211 5 220 n u 332 R \ D 244 265 001 247 0098576 264 C E M \a 035 k 237 \0 242 K J 300 q 3 250 0098592 G 022 022 . 267 312 N 362 h 377 342 350 312 313 220 276 0098608 026 201 260 303 335 241 264 261 266 022 006 a 267 d q i 0098624 017 ! 363 372 a 304 244 236 247 277 Z 367 224 220 G 322 0098640 004 213 375 353 364 270 ( W \n 312 \ 343 E 306 376 177 0098656 341 272 B 321 f ~ 003 032 351 306 017 % < 367 034 017 0098672 002 223 031 C \a 310 307 275 223 340 361 334 0 a 026 p 0098688 r 233 b \f 274 S 4 323 314 366 D 025 003 037 265 250 0098704 213 242 312 212 232 315 305 D g F } 016 f z 223 351 0098720 226 \t 365 275 211 \f 255 \v 325 345 316 ! 274 a Q G 0098736 F 257 035 N ? 361 207 c 251 230 z 245 q y 035 v 0098752 ] 270 221 272 I F 256 . \ 262 245 017 346 236 v C 0098768 I B 036 212 344 ) 277 272 245 - o i 301 [ X \t 0098784 6 351 213 253 365 252 9 327 224 C 035 y 323 + b 364 0098800 9 032 L T 306 K 344 273 374 373 306 275 276 \a ~ 302 0098816 5 r 374 372 207 307 v 374 ! 213 237 336 316 u 324 8 0098832 025 5 325 223 023 252 213 023 8 316 374 177 321 203 x 376 0098848 M ! 242 341 313 277 o I 203 363 224 L 362 347 : 030 0098864 A x 365 324 377 T ' 371 Z 227 207 , 325 364 0098880 224 314 351 266 a < : 277 323 3 235 331 347 204 a w 0098896 002 j H > 243 214 302 241 4 370 \r 321 322 \t 246 035 0098912 p 030 331 307 373 021 214 s y 323 9 [ 027 305 256 i 0098928 314 240 300 @ 312 7 002 O ^ 205 360 254 235 % \t 211 0098944 322 375 367 324 302 211 w 264 243 335 321 w R \r + 276 0098960 - 374 231 274 317 374 / a a K 375 356 243 304 247 H 0098976 H 362 267 235 321 366 ] C 002 N 301 ( 320 ; k h 0098992 270 367 w 234 221 345 317 6 } 363 351 322 032 305 u m 0099008 H 377 215 246 ] 002 / 017 323 314 030 213 ? \n ( 0099024 033 267 343 212 212 304 367 200 331 310 005 275 - | S 237 0099040 1 355 261 C \t 222 327 247 @ 301 i 273 ^ 311 u ; 0099056 d 376 w 206 003 \t 221 261 020 331 P 335 003 \f f 0099072 332 227 212 016 # b P M 354 \a | L V e 035 L 0099088 5 354 343 357 361 m 240 237 N 330 260 D ] 313 > 303 0099104 367 240 002 u q 337 256 250 253 & 2 \ 355 257 K 307 0099120 304 * 232 214 356 327 322 u # P < x 314 271 G 303 0099136 w 304 365 216 021 217 347 257 ? 5 m F > 327 343 ] 0099152 ` 371 \0 V t 221 240 315 035 251 210 270 255 m ( 0099168 003 355 360 207 371 320 303 U 310 } 031 020 b 353 235 325 0099184 ^ 265 n @ 033 202 371 2 245 241 P 372 342 351 1 W 0099200 035 357 025 d $ 327 262 373 = 247 316 X c 310 f \t 0099216 221 217 177 ` 240 250 322 k 236 i 370 f 211 i U 363 0099232 @ 313 214 335 324 307 k 3 u C F i 234 310 Q 253 0099248 267 263 217 343 j I \f . < 204 353 220 l h 355 ; 0099264 351 277 3 275 006 214 324 263 002 3 255 243 > 362 317 334 0099280 021 \ U i 230 # 352 264 , m 306 020 274 204 246 O 0099296 026 206 q 371 # 022 230 343 \v 004 215 214 E C 224 376 0099312 366 W 031 374 036 002 264 215 016 a 374 337 p 240 027 ; 0099328 035 k 375 364 317 256 8 b = 0 337 214 360 [ 375 373 0099344 230 N > 203 364 R 274 257 355 B { I 203 270 373 332 0099360 265 031 332 b 325 214 357 272 I \t m 031 $ 5 273 B 0099376 @ 300 001 f h \f A 252 \r 243 v 327 t 326 5 352 0099392 205 353 366 223 025 277 351 " 207 003 " 203 F 6 323 314 0099408 ( 331 216 232 A [ F 253 205 365 004 Z z 244 225 9 0099424 X 177 _ 205 264 244 W L T 340 204 033 376 i 252 006 0099440 351 . 022 T @ 251 260 \n 231 370 021 ( 332 273 ? ; 0099456 2 245 9 H 7 240 255 F 354 i 334 375 314 301 265 302 0099472 247 241 331 363 x 215 247 327 323 307 351 313 335 1 % s 0099488 234 1 + 327 302 ? 247 ( u $ 360 \a 006 221 017 333 0099504 003 232 z 277 } s 002 w 251 367 310 033 4 201 R 273 0099520 226 e 357 \t # 177 223 q \ 313 351 255 6 323 351 \ 0099536 214 ) l & 342 ; 254 - 243 372 { 301 333 354 \f V 0099552 202 265 C / 211 g " 374 177 < 202 026 205 271 347 253 0099568 A \r ] 022 207 335 365 314 025 337 1 266 247 373 303 321 0099584 ~ 372 # q \f 016 216 u < 207 002 \ 302 x 345 - 0099600 S 316 335 235 344 q v , 300 S < 371 316 335 345 s 0099616 324 276 325 267 363 254 023 , i 257 236 330 n c 352 341 0099632 331 S 313 343 376 B ~ 365 245 305 332 I 203 374 I \a 0099648 361 p = \b \n K 246 017 375 022 0 @ \ 002 324 d 0099664 $ 204 1 - W 324 A 326 ? 215 < ? 025 v S 213 0099680 250 024 023 023 017 d 333 223 / 333 i 327 207 " 363 \r 0099696 R 025 v 343 6 002 , 020 334 \ 231 326 ) 025 316 360 0099712 \t = 362 234 035 \n 277 Y 366 325 M 365 P 346 016 0099728 * w 8 [ 372 s @ U x 202 213 247 [ k 022 021 0099744 311 221 235 024 246 216 = L 370 360 324 316 S 256 n 325 0099760 003 244 035 3 J 232 y 026 311 ; 310 254 N b @ 220 0099776 [ A 027 8 370 221 247 236 036 o & V 217 } 031 275 0099792 255 + 367 206 326 x 001 001 7 L a 253 367 345 : 336 0099808 z > < 355 s 315 313 321 304 325 v 300 255 312 & 0099824 346 221 K f . , 306 ? I \v 210 006 \b N \a z 0099840 195+0 records in 195+0 records out 99840 bytes (100 kB) copied, 0.00459505 s, 21.7 MB/s Nothing here too, I think (certainly not on 097840 or 0098352) Does this help you help me? (Because it does not help me )
November 8, 201213 yr I don;t see any evidence of a reiser file system. (and that is why unRAID cannot mount the disk) The MBR shows your partition starts on sector 63. If this is an older disk that is correct. Is there any chance you used the 4k-aligned option when you originally installed it? If you did, the file-system should have been at the 0098352 address. Since it is at neither, all you can do is try to rebuild the superblock. You could not un-mount /dev/md9, since it was never mounted. (and why it showed as unformatted) You can try reiserfsck --rebuild-sb /dev/md9 and see if it can rebuild the superblock. See here for the options to be used: http://lime-technology.com/forum/index.php?topic=1483 Joe L.
November 8, 201213 yr Author Well, the drive is a WD 2TB EARS and it has a jumper on it which means I installed it before unraid 4.6 and according to Unraid it is unaligned. I will proceed to rebuilding the superblock. But, before I make any permanent changes, is it possible to use any tool that locates lost files in it? (outside the scope of Unraid). I know that there are some such tools in windows but I would think they work for Disks that are (or were) NTFS. Also, since it does not see a /dev/md9, maybe instead of reiserfsck --rebuild-sb /dev/md9 I should run reiserfsck --rebuild-sb /dev/sdj1?
November 8, 201213 yr Well, the drive is a WD 2TB EARS and it has a jumper on it which means I installed it before unraid 4.6 and according to Unraid it is unaligned. I will proceed to rebuilding the superblock. But, before I make any permanent changes, is it possible to use any tool that locates lost files in it? (outside the scope of Unraid). I know that there are some such tools in windows but I would think they work for Disks that are (or were) NTFS. If you have an equal sized disk as a spare, you can use dd if=/dev/sdj of=/dev/sdX bs=1M where sdX = the device name of the spare disk to copy the disk in its entirety. Just be very careful, as disk device names change from one boot to the next. Make sure the failed disk is still /dev/sdj, and the target disk is the one not assigned to the array. Use ls -l /dev/disk/by-id to confirm the model/serial number against the device names. There are some utilities to recover files, I've not got any info, I've not used them. I would clone the disk, then perform the recovery efforts on the cloned copy. Joe L.
November 8, 201213 yr Also, since it does not see a /dev/md9, maybe instead of reiserfsck --rebuild-sb /dev/md9 I should run reiserfsck --rebuild-sb /dev/sdj1? You can certainly try that. If you do, parity will not be maintained, but if it gets you your data, you can correct parity.
November 9, 201213 yr Author I am happy to report that everything worked and with the help of Joe (to whom a 'thank you' worth 2TBs of data is due). I will summarize a bit for any unlucky soul that encounters my problem: 1. I checked the ReiserFS using the methods described by Joe in a previous post of this thread. No evidence of ReiserFS was found (why? I don't know what happened). 2. Since I found no evidence of ReiserFS I proceeded to run reiserfsck --rebuild-sb /dev/sdj1 in order to rebuild superblock. That took a long time. 3. That wasn't enough so I proceeded to run reiserfsck --rebuild-tree /dev/sdj1 in order to force fsck to rebuild filesystem from scratch. That also took a long time but I was happily watching files I thought were lost being discovered. 4. Since I changed and fixed things in the disk, I run a new parity check. It is obviously finding lots of errors, which it is currently fixing. If a disk does not fail, in 12 hours I will be parity-protected. The only remaining problem is that I never found why this problem occurred. I hope it is not an unraid bug because I must admit my faith in it has been a bit shaken.
Archived
This topic is now archived and is closed to further replies.