Hello,

I try to explain my weird problem I noticed yesterday:

 

. 3 weeks ago I've got 768 write errors on Disk4 therefore I've decided to replace it with a new one (UnRAID 5.0 RC13).

. Once replaced I started the rebuild process that took 6 hours and after that I've had the Array once again green and without errors.

. I used it for some times without seeing nothing strange so I decided to update it to the 5.0 stable.

. The upgrade process ran smoothly and after that I've decided to transfer some files onto it and here the nightmare began.  :'(

. One folder on my SMB share was competely empty if accessed like this \\unraid\share\folder1\ completely empty but if I try to access it pointing directly to the disk share a great part of the contained files was there. \\unraid\disk4\folder1\ most of the content is there.

. But not all the files that I had was there: I expect 27 files but only 24 was there!!!

. The first thing I've thinked was a permission problem so I decided to access my Server through SSH and effectively, after browsing to the Disk4 mount, I saw that each 27 files are there but 3 of them where missing the permissions.

. I finally decided to reset all the permissions through the New Permission function but, after running smoothly beetween the first 3 disks, it stopped on Disk 4.

. Very strange, this is a new disk, how can be possible that I have two problems on the same disk4 even if it's a new one?

. So I decided to change the SATA port to ensure that the cable or the SATA port was not creating this problem but the New Permission continues to stop on Disk4.

. Why? The answer lies on the Disk4 that now is marked as Read Only FS and onto an excerpt of my Syslog:

 

Oct  4 00:28:29 MegaStore kernel: REISERFS (device md4): checking transaction log (md4)

Oct  4 00:28:29 MegaStore kernel: REISERFS (device md4): Using r5 hash to sort names

Oct  4 00:28:29 MegaStore emhttp: shcmd (33): chmod 777 '/mnt/disk4'

Oct  4 00:28:29 MegaStore emhttp: _shcmd: shcmd (33): exit status: 1

Oct  4 00:28:29 MegaStore emhttp: shcmd (34): chown nobody:users '/mnt/disk4'

Oct  4 00:28:29 MegaStore emhttp: _shcmd: shcmd (34): exit status: 1

Oct  4 00:28:29 MegaStore emhttp: shcmd (35): mkdir /mnt/disk5

Oct  4 00:28:29 MegaStore emhttp: shcmd (36): set -o pipefail ; mount -t reiserfs -o user_xattr,acl,noatime,nodiratime /dev/md5 /mnt/disk5 |& logger

Oct  4 00:28:29 MegaStore kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 36758 does not match to the expected one 3

Oct  4 00:28:29 MegaStore kernel: REISERFS error (device md4): vs-5150 search_by_key: invalid format found in block 190757216. Fsck?

Oct  4 00:28:29 MegaStore kernel: REISERFS (device md4): Remounting filesystem read-only

Oct  4 00:28:29 MegaStore kernel: REISERFS error (device md4): vs-2140 finish_unfinished: search_by_key returned -2

Oct  4 00:28:29 MegaStore avahi-daemon[1608]: Server startup complete. Host name is MegaStore.local. Local service cookie is 2563017304.

Oct  4 00:28:29 MegaStore kernel: REISERFS (device md5): found reiserfs format "3.6" with standard journal

Oct  4 00:28:29 MegaStore kernel: REISERFS (device md5): using ordered data mode

Oct  4 00:28:29 MegaStore kernel: reiserfs: using flush barriers

Oct  4 00:28:29 MegaStore kernel: REISERFS (device md5): journal params: device md5, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30

Oct  4 00:28:29 MegaStore kernel: REISERFS (device md5): checking transaction log (md5)

Oct  4 00:28:29 MegaStore kernel: REISERFS (device md5): Using r5 hash to sort names

Oct  4 00:28:30 MegaStore emhttp: shcmd (37): chmod 777 '/mnt/disk5'

Oct  4 00:28:30 MegaStore emhttp: shcmd (38): chown nobody:users '/mnt/disk5'

Oct  4 00:28:30 MegaStore emhttp: shcmd (39): mkdir /mnt/disk6

Oct  4 00:28:30 MegaStore emhttp: shcmd (40): set -o pipefail ; mount -t reiserfs -o user_xattr,acl,noatime,nodiratime /dev/md6 /mnt/disk6 |& logger

Oct  4 00:28:30 MegaStore kernel: REISERFS (device md6): found reiserfs format "3.6" with standard journal

Oct  4 00:28:30 MegaStore kernel: REISERFS (device md6): using ordered data mode

Oct  4 00:28:30 MegaStore kernel: reiserfs: using flush barriers

Oct  4 00:28:30 MegaStore kernel: REISERFS (device md6): journal params: device md6, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30

Oct  4 00:28:30 MegaStore kernel: REISERFS (device md6): checking transaction log (md6)

Oct  4 00:28:30 MegaStore kernel: REISERFS (device md6): Using r5 hash to sort names

Oct  4 00:28:30 MegaStore emhttp: shcmd (41): chmod 777 '/mnt/disk6'

Oct  4 00:28:30 MegaStore emhttp: shcmd (42): chown nobody:users '/mnt/disk6'

Oct  4 00:28:30 MegaStore emhttp: shcmd (43): mkdir /mnt/user

Oct  4 00:28:30 MegaStore emhttp: shcmd (44): /usr/local/sbin/shfs /mnt/user -disks 16777214 -o noatime,big_writes,allow_other -o remember=0  |& logger

Oct  4 00:28:30 MegaStore emhttp: shcmd (45): crontab -c /etc/cron.d -d &> /dev/null

Oct  4 00:28:30 MegaStore emhttp: shcmd (46): /usr/local/sbin/emhttp_event disks_mounted

Oct  4 00:28:30 MegaStore emhttp_event: disks_mounted

Oct  4 00:28:30 MegaStore emhttp: shcmd (47): :>/etc/samba/smb-shares.conf

Oct  4 00:28:30 MegaStore emhttp: Restart SMB...

Oct  4 00:28:30 MegaStore emhttp: shcmd (48): killall -HUP smbd

Oct  4 00:28:30 MegaStore emhttp: shcmd (49): ps axc | grep -q rpc.mountd

Oct  4 00:28:30 MegaStore emhttp: _shcmd: shcmd (49): exit status: 1

Oct  4 00:28:30 MegaStore emhttp: shcmd (50): /usr/local/sbin/emhttp_event svcs_restarted

Oct  4 00:28:30 MegaStore emhttp_event: svcs_restarted

Oct  4 00:28:30 MegaStore avahi-daemon[1608]: Service "MegaStore-SMB" (/services/smb.service) successfully established.

Oct  4 00:31:22 MegaStore kernel: REISERFS warning: reiserfs-5089 is_internal: free space seems wrong: level=3, nr_items=116, free_space=832 rdkey

Oct  4 00:31:22 MegaStore kernel: REISERFS error (device md4): vs-5150 search_by_key: invalid format found in block 179995974. Fsck?

Oct  4 00:31:22 MegaStore kernel: REISERFS error (device md4): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [23205 23206 0x0 SD]

Oct  4 00:31:22 MegaStore kernel: REISERFS warning: reiserfs-5089 is_internal: free space seems wrong: level=3, nr_items=116, free_space=832 rdkey

Oct  4 00:31:22 MegaStore kernel: REISERFS error (device md4): vs-5150 search_by_key: invalid format found in block 179995974. Fsck?

Oct  4 00:31:22 MegaStore kernel: REISERFS error (device md4): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [23205 23222 0x0 SD]

Oct  4 00:31:22 MegaStore kernel: REISERFS warning: reiserfs-5089 is_internal: free space seems wrong: level=3, nr_items=116, free_space=832 rdkey

Oct  4 00:31:22 MegaStore kernel: REISERFS error (device md4): vs-5150 search_by_key: invalid format found in block 179995974. Fsck?

Oct  4 00:31:22 MegaStore kernel: REISERFS error (device md4): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [23205 23225 0x0 SD]

Oct  4 00:31:22 MegaStore kernel: REISERFS warning: reiserfs-5082 is_leaf: free space seems wrong: level=1, nr_items=2, free_space=1220 rdkey

Oct  4 00:31:22 MegaStore kernel: REISERFS error (device md4): vs-5150 search_by_key: invalid format found in block 220037076. Fsck?

Oct  4 00:31:22 MegaStore kernel: REISERFS error (device md4): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [309 30416 0x0 SD]

Oct  4 00:31:23 MegaStore kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 36758 does not match to the expected one 3

Oct  4 00:31:23 MegaStore kernel: REISERFS error (device md4): vs-5150 search_by_key: invalid format found in block 190757216. Fsck?

Oct  4 00:31:23 MegaStore kernel: REISERFS error (device md4): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [61271 61272 0x0 SD]

Oct  4 00:31:23 MegaStore kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 36758 does not match to the expected one 3

Oct  4 00:31:23 MegaStore kernel: REISERFS error (device md4): vs-5150 search_by_key: invalid format found in block 190757216. Fsck?

Oct  4 00:31:23 MegaStore kernel: REISERFS error (device md4): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [61261 61263 0x0 SD]

Oct  4 00:31:23 MegaStore kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 36758 does not match to the expected one 3

Oct  4 00:31:23 MegaStore kernel: REISERFS error (device md4): vs-5150 search_by_key: invalid format found in block 190757216. Fsck?

Oct  4 00:31:23 MegaStore kernel: REISERFS error (device md4): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [61261 61268 0x0 SD]

Oct  4 00:31:23 MegaStore kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 36758 does not match to the expected one 3

Oct  4 00:31:23 MegaStore kernel: REISERFS error (device md4): vs-5150 search_by_key: invalid format found in block 190757216. Fsck?

Oct  4 00:31:23 MegaStore kernel: REISERFS error (device md4): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [61261 61262 0x0 SD]

 

. Fortunatly I have a backup of the entire Disk4 (done on the old Disk4) but right now I have no idea how to restore the normal situation and how to prevent the problem from occurring once again. Can it be possible that a Filesystem corruption on one disk spreads over the Parity Disk and after that, even if I replace the faulted disk, the rebuild process creates once again a corrupted filesystem over and over?

 

Thanx in advance for your support.

 

Luke

A disk rebuild will not correct file system errors. See Check Disk Filesystem in my sig.

Thanx a lot,

I've followed the instruction to repair the file system corruption and in the end I've done a new config to reconstruct the parity disk and repair the weird SMB behaviour. Now the New Permissions has worked like a charm and solved all the permission problems. Thanx again for your support!

 

Best regards.

 

Luca