Folks,

 

I am having an issue when I try to start the array. The server crashes with the following error on the console:

 

Kernel panic - not syncing: <4>REISERFS panic: vs-6030 check_internal_block_head: invalid item number level=2, nr_items=253, free_space=63528 rdkey

 

I did a search on kernel panic here and found that the problem is most likely related to power supply. So, I tried 3 different power supplies, but the problem still persists. I would really appreciate some help. Thanks!

 

I am also attaching the syslogs up untill the point it crashes.

syslogs.txt

It seems like you have a corrupt filesystem. You should proceed with troubleshooting filesystem corruptions. Look for one of the many excellent posts by Joe L. on this subject.

Thanks. I tried to follow the instructions specified here: http://lime-technology.com/wiki/index.php?title=Check_Disk_Filesystems

 

But, 'reiserfsck --check /dev/md1' fails because it is not present in my system. Should I instead replace md5 with the sda, sdb, hda, hdb so on and so forth and continue?

 

Thanks.

Thanks. I tried to follow the instructions specified here: http://lime-technology.com/wiki/index.php?title=Check_Disk_Filesystems

 

But, 'reiserfsck --check /dev/md1' fails because it is not present in my system. Should I instead replace md5 with the sda, sdb, hda, hdb so on and so forth and continue?

 

Thanks.

If the "md" device is not being initially loaded, the "md1, md2, md3" devices will not be there.

 

You'll instead need to use the "first" partition of each data disk.  (Note the trailing "1" on the device partition being checked.)

 

reiserfsck --check /dev/sda1

or

reiserfsck --check /dev/sdb1

or

reiserfsck --check /dev/hda1

etc...

 

Remember, the parity drive has no file-system, so do not attempt to check it, you'll only corrupt what is there.

Also, once you perform any repair on any file-system, the parity will be invalid. It is no longer in sync since you worked on the raw drives instead of the "md" devices.  When you next bring the array on-line, you MUST perform a parity check, and it will find "errors" (differences) based on your file-system repairs.  A second subsequent parity check should then find no errors.

 

 

Thank Joe L.

 

Found problems with the two 1.5TB Samsung drives. Here is the output from the reiserfsck:

 

Replaying journal: Done.
Reiserfs journal '/dev/sda1' in blocks [18..8211]: 0 transactions replayed
Checking internal tree.. \/  1 (of   8|/  3 (of 107// 65 (of 170-bad_path: block                                              226224526, pointer 64: The used space (1272) of the child block (226250602) is                                              not equal to the (blocksize (4096) - free space (2768)/ 39 (of 107//170 (of 170\                                             bad_path: The right delimiting key [6531 6918 0x31d001 IND (1)] of the node (329                                             030322) must be greater than the last (2) element's key [6531 6990 0x1 IND (1)]                                              within the node.                                      / 40 (of 107|/  1 (of 170/                                             bad_node: vpf-10350: The block (328953447) is used more than once in the tree.
the problem in the internal node occurfinished                                /
Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs.
Bad nodes were found, Semantic pass skipped
2 found corruptions can be fixed only when running with --rebuild-tree
###########
reiserfsck finished at Sat Aug  7 20:41:07 2010
###########

###########
reiserfsck --check started at Sat Aug  7 20:42:36 2010
###########
Replaying journal: Done.
Reiserfs journal '/dev/sdc1' in blocks [18..8211]: 0 transactions replayed
Checking internal tree.. \/  3 (of  17\/158 (of 158|/170 (of 170-bad_path: The right delimiting key [1450 1472 0x4d9001 IND (1)] of the node (40852458) must be greater than the last (0) element's key [1519 1522 /  4 (of  17\/  1 (of 157|/  1 (of 170/bad_path: The left delimiting key [1450 1472 0x4d9001 IND (1)] of the node (40852459) must be equal to the first element's key [1519 1522 0x5ef2f001 IND (1)] within the node.                                     / 13 (of  17\/152 (of 167|/137 (of 149/block 105561322: The level of the node (25065) is not correct, (1) expected
the problem in the internal node occured (105561322), whole subtree is skipped       / 14 (of  17\/ 30 (of 170\/ 73 (of  86|bad_stat_data: The objectid (432117) is shared by at least two files. Can be fixed wifinished
Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs.
Bad nodes were found, Semantic pass skipped
3 found corruptions can be fixed only when running with --rebuild-tree
###########
reiserfsck finished at Sat Aug  7 20:56:17 2010
###########

 

Please advise if I should go ahead and run it with --rebuild-tree option on those two drives. Also, what are the chances of being able to recover majority, if not all the data?

 

Thanks!

 

Experience has been that most will be recoverable.

 

One thing for certain, if you do nothing, you can't recover anything.

 

Let "reiserfsck" guide you.  do only what it suggests.

 

A FAR bigger question in my mind is what caused the corruption.  For it to have occurred to two drives I'd be looking for a common cause.

(perhaps a power failure or a bad power supply, or bad memory, or memory with the wrong voltage, timing, or clock speed.)

Joe, Thanks a lot for your help. Appreciate it a lot.

 

The recovery went fine and it is presently performing a parity sync.

 

A FAR bigger question in my mind is what caused the corruption.  For it to have occurred to two drives I'd be looking for a common cause.

(perhaps a power failure or a bad power supply, or bad memory, or memory with the wrong voltage, timing, or clock speed.)

 

I have to mention that there where some existing issues with those drives which lead up to this problem. If I tried to save any file into those two drives, the system would crash. But there was no problem reading files. It did not occur to me that it could be the result of a data corruption. So I was trying all sorts of other things like changing power supplies, making sure all the connections where in good shape, etc while in fact, I should have done a file system check. 

 

There is one other issue I see in the syslogs related to the USB flash drive:

 

Aug 7 18:17:56 Tower kernel: md: unRAID driver 0.95.4 installed
Aug 7 18:17:56 Tower kernel: FAT: Filesystem error (dev sde1)
Aug 7 18:17:56 Tower kernel: fat_get_cluster: invalid cluster chain (i_pos 998282)
Aug 7 18:17:56 Tower kernel: File system has been set read-only
Aug 7 18:17:56 Tower kernel: FAT: Filesystem error (dev sde1)
Aug 7 18:17:56 Tower kernel: fat_get_cluster: invalid cluster chain (i_pos 998282)
Aug 7 18:17:56 Tower kernel: FAT: Filesystem error (dev sde1)
Aug 7 18:17:56 Tower kernel: fat_get_cluster: invalid cluster chain (i_pos 998282)
Aug 7 18:17:56 Tower kernel: FAT: Filesystem error (dev sde1)
Aug 7 18:17:56 Tower kernel: fat_get_cluster: invalid cluster chain (i_pos 998282)
Aug 7 18:17:56 Tower kernel: FAT: Filesystem error (dev sde1)
Aug 7 18:17:56 Tower kernel: fat_get_cluster: invalid cluster chain (i_pos 998282)
Aug 7 18:17:56 Tower kernel: FAT: Filesystem error (dev sde1)
Aug 7 18:17:56 Tower kernel: fat_get_cluster: invalid cluster chain (i_pos 998282)
Aug 7 18:17:56 Tower kernel: FAT: Filesystem error (dev sde1)
Aug 7 18:17:56 Tower kernel: fat_get_cluster: invalid cluster chain (i_pos 998282)
Aug 7 18:17:56 Tower kernel: FAT: Filesystem error (dev sde1)
Aug 7 18:17:56 Tower kernel: fat_get_cluster: invalid cluster chain (i_pos 998282)
Aug 7 18:17:56 Tower kernel: FAT: Filesystem error (dev sde1)
Aug 7 18:17:56 Tower kernel: fat_get_cluster: invalid cluster chain (i_pos 998282)
Aug 7 18:17:56 Tower kernel: read_file: read error 5

 

And I got this error when I started the latest parity sync:

 

Aug 8 13:01:22 Tower kernel: write_file: error 30 opening /boot/config/super.dat

 

I see that the /boot partition is mounted as a read only file system. Any suggestions on how to fix this?

 

Thanks!

I think something is also wrong with your USB flash drive.

I think something is also wrong with your USB flash drive.

Exactly.

 

You need to move it to your PC and run a scandisk/chkdisk on it.  Since the file-system on the flash drive is corrupted, it is being mounted as read-only to prevent further file corruption.

Can I move it to a PC, fix it and move it back, while the system is running or should I wait for the parity sync to complete and then shutdown the system before I move it. The reason I ask is because if the system is not able to write to /boot/config/super.dat, would the array be invalid again when the system boot back up after I fix the flash drive on the PC? That would involve building the parity all over again.

 

Thanks!

Can I move it to a PC, fix it and move it back, while the system is running or should I wait for the parity sync to complete and then shutdown the system before I move it. The reason I ask is because if the system is not able to write to /boot/config/super.dat, would the array be invalid again when the system boot back up after I fix the flash drive on the PC? That would involve building the parity all over again.

 

Thanks!

Move it now.  Then move it back and reboot.

 

As you said, it will not be able to record the successful parity check anyway until it is fixed and mounted as normal.