All,

 

I'm not entirely sure when/where the problem(s) began.  The first thing I noticed is that I had some SQL errors in SAB running on my cache drive.  Nothing significant, the system could continue to run but with continual "warning messages".  Then this morning, I went to move a file from the cache to one of my shares (TV), and apparently I goofed up. (mv file.ext /mnt/cache/TV with no final "/")  After that, I could no longer access that share.  The other shares were fine and seem unaffected.  So I tried a restart of the system, but no luck.  Again, the other shares were fine but now the system didn't even list a TV share (it previously listed it, but I couldn't access it). 

 

So I put the system in maintenance mode and starting running reiserfsck --check on the disks in my array (1-5), and #4 found a problem.  (See attached log for md4) 

 

bad_directory_item: block 670864158: The directory item [19458 19470 0x53bcf980 DIR (3)] has a not properly hashed entry (10)

bad_leaf: block 670864158, item 0: The corrupted item found (19458 19470 0x53bcf980 DIR (3), len 2464, location 1632 entrfinished

Comparing bitmaps..finished

Fatal corruptions were found, Semantic pass skipped

1 found corruptions can be fixed only when running with --rebuild-tree

 

Additionally, I ran reiserfsck --check on my cache drive (/dev/sde1) and found problems: (see attached log for sde1)

The left delimiting key [165220 165189 0x0 SD (0)] of the node (45763641) must be equal to the first element's key [165188 165189 0x0 SD (0)] within the node.                                                  /  5 (of  7//120 (of 131|/ 34 (of 170-block 115692534: The level of the node (50743) is not correct, (1) expected

the problem in the internal node occured (115692534), whole subtree is skipped                                          /  7 (of  7-/ 79 (of 114|/161 (of 170/bad_indirect_item: block 83081075: The item (800034 801871 0xbb69001 IND (1), len 4048, location 48 entry count 0, fsck need 0, format new) has the bad pointer (974) to the block (83084093), which is in tree already                                                                  finished

Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs.

Bad nodes were found, Semantic pass skipped

2 found corruptions can be fixed only when running with --rebuild-tree

 

I have syslogs from before the restart and after.

 

Should I follow the reiserfsck recommends to rebuild-tree?  What is wrong with my system?  Should I "re-add" a share called "TV"?

 

Thanks!

Reiserfsck_-md4.txt

Reiserfsck_-sde1.txt

syslogs.zip

Since its telling you to run --rebuild-tree, you should

 

But, from looking over your after-reboot syslog, you do have some other issues.

 

It looks like both ST3000DM001-1CH166 W1F4YW2K and ST2000DM001-1CH164 Z340K6EA (disk 2 and 5) may either have a loose cable (try reseating them) or your power supply is weak, as they are both spitting out errors that usually indicates cabling / power.  Power off the system and reseat all the sata cables and power connectors to those drives and at the controller (wouldn't be a bad idea to also reseat the controller if its an add-on card)

 

 

Thanks Squid.  I have had a connector issue in the past (it was only half on) so I'll double check, and I do have a SATA card so I'll check that as well.  Power supply should be ok, its a rather new Seasonic 650W unit (http://www.newegg.com/Product/Product.aspx?Item=N82E16817151118) with 54A on the single 12V rail.

 

Any preference the order in which I run "rebuild tree"? md4 first, then cache? Vice versa? (Not having run it before, I don't know what to expect.  Just going off the guidance from the Check Disk Filesystems Wiki page).

Any preference the order in which I run "rebuild tree"? md4 first, then cache? Vice versa? (Not having run it before, I don't know what to expect.  Just going off the guidance from the Check Disk Filesystems Wiki page).

You'll have to ask the magic 8 ball for that answer    Doesn't make a difference.

 

Depending on how many files, etc it could take a while to run.  I would do it at a console instead of through telnet / putty (just in case your computer resets / goes to sleep, etc)

Squid, the rebuild-tree has finished on both drives.  I stopped the array, and started back up outside of Maintenance mode.  All the other shares came back, but still no "TV" share.  Should I restart?  Do I need to "readd" the share?

I'm not exactly sure why you wouldn't be able to access your TV share anymore.  Since that folder is on your cache, you can safely delete it from the cache without affecting your array.  (but don't delete your user share)  You may have to recreate the folder.

 

If I do something like mv filename.ext /mnt/cache/tv (without the ending /) it successfully moves the file into the tv folder the same as if I had the /

 

What does the shares tab in the gui show?

Just did a full reboot (it had trouble unmounting disks so I had to use the "powerdown" command), but it's up again and it found the TV share again.  Everything looks like it's back to normal.

 

That being said, would you mind review my syslog to see if there are anything issues (you mentioned the HDD issue, but I don't see any msgs related to it anymore).  I've attached the syslog from after the latest restart.

syslog-2015-03-02.txt

Looks good

Thanks!  Truly appreciate the help!