unable to run reiserfsck on drive with drive with read_locked_inode - reiserfsck - bread: End of file, cannot read the block (3906469874)


Recommended Posts

I have an "old-faithful" version of UnRaid (5.0-rc8a) - been running great for 8 years :-| If it ain't broke ... :-) It's stock .. nothing but "Simple Features" installed - just used for file storage

 

I had issues copying files to my newest drive (18TB WD Ultrastar DC HC550) and it reported as being in "read-only" mode.

I presumed it was working as I upgraded an old 10TB drive and it all seemed AOK - content seemed ok.

 

Checking recent logs showed this

Mar 8 13:14:23 TowerToo kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one 1 Mar 8 13:14:23 TowerToo kernel: REISERFS error (device md10): vs-5150 search_by_key: invalid format found in block 1836494317. Fsck? Mar 8 13:14:23 TowerToo kernel: REISERFS (device md10): Remounting filesystem read-only Mar 8 13:14:23 TowerToo kernel: REISERFS error (device md10): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [4 1126 0x0 SD] Mar 8 13:14:23 TowerToo kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one 1 Mar 8 13:14:23 TowerToo kernel: REISERFS error (device md10): vs-5150 search_by_key: invalid format found in block 1836494317. Fsck? Mar 8 13:14:23 TowerToo kernel: REISERFS error (device md10): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [4 1123 0x0 SD] Mar 8 13:14:35 TowerToo kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one 1 Mar 8 13:14:35 TowerToo kernel: REISERFS error (device md10): vs-5150 search_by_key: invalid format found in block 1836494317. Fsck? Mar 8 13:14:35 TowerToo kernel: REISERFS error (device md10): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [4 1126 0x0 SD] Mar 8 13:14:35 TowerToo kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one 1 Mar 8 13:14:35 TowerToo kernel: REISERFS error (device md10): vs-5150 search_by_key: invalid format found in block 1836494317. Fsck? Mar 8 13:14:35 TowerToo kernel: REISERFS error (device md10): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [4 1123 0x0 SD] Mar 8 13:14:50 TowerToo kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one 1 Mar 8 13:14:50 TowerToo kernel: REISERFS error (device md10): vs-5150 search_by_key: invalid format found in block 1836494317. Fsck? Mar 8 13:14:50 TowerToo kernel: REISERFS error (device md10): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [4 1126 0x0 SD] Mar 8 13:14:50 TowerToo kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one 1 Mar 8 13:14:50 TowerToo kernel: REISERFS error (device md10): vs-5150 search_by_key: invalid format found in block 1836494317. Fsck? Mar 8 13:14:50 TowerToo kernel: REISERFS error (device md10): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [4 1123 0x0 SD]

 

 

So I attempted to follow the instructions here https://wiki.unraid.net/Check_Disk_Filesystems#Is_this_the_right_page.3F

 

Which said ID your OS - 5.0 rc8a which meant that I had the UnRAID 5 process to follow

 

That said to stop array and start in Mainenance mode - which I did AOK.

 

Telnet in AOK

 

df -h

 

 

root@TowerToo:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdi1              15G   61M   15G   1% /boot
/dev/md1               11T  7.6T  3.4T  70% /mnt/disk1
/dev/md2              9.1T  8.9T  201G  98% /mnt/disk2
/dev/md3              9.1T  9.0T  168G  99% /mnt/disk3
/dev/md4              9.1T  9.0T  170G  99% /mnt/disk4
/dev/md5              9.1T  9.0T  158G  99% /mnt/disk5
/dev/md6              9.1T  9.1T   84G 100% /mnt/disk6
/dev/md7              9.1T  9.0T  180G  99% /mnt/disk7
/dev/md8              9.1T  9.0T  188G  98% /mnt/disk8
/dev/md9              9.1T  8.6T  572G  94% /mnt/disk9
/dev/md10              15T  7.0T  7.7T  48% /mnt/disk10
/dev/md11              11T  8.5T  2.5T  78% /mnt/disk11
shfs                  110T   95T   16T  87% /mnt/user

 

Issues with this are that it identifies the drive sizes wrong -

/dev/md10 is 18TB

even with creative Math for TiB and TB ..I'd still expect 16 - 17TB as listed.

 

fdisk -l shows it's correct

 

Disk /dev/sdm: 18000.2 GB, 18000207937536 bytes
256 heads, 63 sectors/track, 2179852 cylinders
Units = cylinders of 16128 * 512 = 8257536 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

 

Moving on though ..

 

When I attempt to reiserfsck

 

It seems to instantly have an issue ...

 

root@TowerToo:~# reiserfsck --check /dev/md10
reiserfsck 3.6.21 (2009 www.namesys.com)

*************************************************************
** If you are using the latest reiserfsprogs and  it fails **
** please  email bug reports to [email protected], **
** providing  as  much  information  as  possible --  your **
** hardware,  kernel,  patches,  settings,  all reiserfsck **
** messages  (including version),  the reiserfsck logfile, **
** check  the  syslog file  for  any  related information. **
** If you would like advice on using this program, support **
** is available  for $25 at  www.namesys.com/support.html. **
*************************************************************

Will read-only check consistency of the filesystem on /dev/md10
Will put log info to 'stdout'

Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes

 

 

bread: End of file, cannot read the block (3906469874).

Aborted

 

Does this mean:

- it can't find the drive at all?

- drive is corrupted?

- perhaps reiserfsck is too old for this drive (too large?) 3.6.21 and 18 TB drive?

 

If it's too old ... would upgrading to UnRAID 6.X series be beneficial? Or will it (reasonably) object to trying to upgrade a mangled Array?

 

Any helpful suggestions welcomed at this point.

 

Longer term I'm looking to building a completely new UnRAID server ... just not now as it's not a rapid process choosing components, finding $$$ and then building it (ie migrating across content).
 

 

Screen Shot 2021-03-10 at 11.24.00 pm.png

Link to comment

Is the maximum size of ReiserFS 16 TiB?

https://reiser4.wiki.kernel.org/index.php/FAQ_ReiserFS#What_are_the_specs_for_ReiserFS:_maximum_number_of_files.2C_of_files_a_directory_can_have.2C_of_sub-dirs_in_a_dir.2C_of_links_to_a_file.2C_maximum_file_size.2C_maximum_filesystem_size.2C_etc..3F

max filesystem size232 (4K) blocks => 16 Ti232 (4K) blocks => 16 Ti

 

If that's the issue ... can I get a 16TB drive from somewhere ... (buy 😐 ) ... rebuild with that parity ... essentially swapping out a large drive for a smaller one ... but sorta not .. as Reiser only thought it was 15TiB?

 

And then the issues will go away ... and *then* upgrade to UnRAID 6.X to get larger filesystem support ?

Link to comment
On 3/10/2021 at 11:57 PM, JorgeB said:

According to wiki max Reiser FS volume size is 16TiB, so likely why you're running into trouble, best to upgrade to Unraid V6 and convert to XFS.

I have been attempting to upgrade my unRAID 5.0-rc8a as per the instructions here https://wiki.unraid.net/UnRAID_6/Upgrade_Instructions

Which pointed me to here

 

 

which led to the first challenge ...

 

wget --no-check-certificate https://raw.githubusercontent.com/limetech/unRAIDServer/master/unRAIDServer.plg

 

doesn't actually work due to the age of the wget command (doesn't support newer encryption standards)

 

and hangs ... but as curl is available ... I was able to grab the file

 

I had a look to see if there were any other wget incompatible URLs

 

and it seems that it's not possible to download the versions of unRAID linked in the plugin - perhaps because they are all too old? The post was from 2018 and it's 2021

 

All of these get 403s

 

pluginURL "https://s3.amazonaws.com/dnld.lime-technology.com/stable/unRAIDServer.plg"

zip "https://s3.amazonaws.com/dnld.lime-technology.com/stable/unRAIDServer-6.5.3-x86_64.zip"

md5 "https://s3.amazonaws.com/dnld.lime-technology.com/stable/unRAIDServer-6.5.3-x86_64.md5"

 

Clutching at straws I tried with CURL but I got the same 403 not authorised as expected 😐

 

curl --insecure http://slackware.cs.utah.edu/pub/slackware/slackware-13.1/slackware/a/infozip-6.0-i486-1.txz -o infozip-6.0-i486-1.txz

curl --insecure https://s3.amazonaws.com/dnld.lime-technology.com/stable/unRAIDServer-6.5.3-x86_64.zip -o unRAIDServer.zip

curl --insecure https://s3.amazonaws.com/dnld.lime-technology.com/stable/unRAIDServer-6.5.3-x86_64.md5 -o unRAIDServer.md5

 

All files have content like this

 

cat unRAIDServer.zip
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>KTBC2NBJWMNYWCYB</RequestId><HostId>8ouBTGUTSAaYiIi30HqQ/3JaRj9oFrQnynQGVZfmFJxtr68CJ/sHPvL/q//rAZLKYEQy2/rtwlw=</HostId></Error>

 

 

Any thoughts on how I can proceed from here?

Can I just drop in the latest version 6.9.1 instead?

https://s3.amazonaws.com/dnld.lime-technology.com/stable/unRAIDServer.plg

 

 

 

 

Link to comment

Some progress on getting myself a viable unRAID system.

 

I tried the option 1 approach - as I indicated - just changing the version in the plg file.

 

That stated that it worked properly without issues. But it failed to startup.

 

I had the "Could not find kernel image: linux"

 

Which I was able to fix by following the instructions here

ie copy bzroot, bzimage, syslinux and made it bootable.

That gave a running instance so I ran the "new permissions" tool ...

 

which took a long time and "seemed" to pass AOK ...

 

But none of my user shares are present (only the actual "disk" ones).

I tried to manually create one of the shares ... and I did so .. but it failed to work.

 

I did notice down the bottom of the unRAID page that it stated

 

"Array Started - Starting services"

 

Starting ... vs Started ... in progress.

 

Then I saw the logs were full of errors for the too large ReisterFS drive

 

Mar 15 11:56:57 TowerToo emhttpd: error: get_filesystem_status, 6099: Permission denied (13): scandir Permission denied

 

 

Open to suggestions as to how I can proceed from here?

 

The obvious option that I can think of is to replace the ReiserFS drive with an XFS one ... but that takes time to get a new drive ( I have ordered one).
Is it likely that will work? ie .. swapping out the ReiserFS 18 TB drve (with 16TB of limited by ReiserFS content) ... letting it rebuild using Parity Data onto an XFS formatted drive?

 

And given that the NewPermissions doesn't appear to work .. I suspect that means I need to run them again on the new 18TB drive should it successfully rebuild (the parity is from before the Permissions).

 

thanks for listening :)

 

 

Link to comment

I have 12 SATA ports ... and 11 data disks and 1 parity disk ... so there is no create a new drive and copy across the contents.

There is insufficient space upon the remaining 10 drives to move the content off the 18 TB drive.

 

So the only options that I see are:

  • add another drive via USB - which I suspect isn’t viable (I have not found USB anything to be anywhere near reliable enough for a RAID)
  • copy off the content to another server
    • then wipe the 18TB drive and reformat it as XFS and copy back the content

I have never tried to “remove and reformat” a drive in UNRAID. How do I go about reducing the capacity of my total array?

 

thanks again for helping @JorgeB

Link to comment
1 hour ago, JorgeB said:

After you clear the largest disk you just need to click on it, change filesystem to xfs, format and restore the data.

 

After I have copied all the content from the largest drive to somewhere appropriate ...

 

I can then click on the 18 TB data drive 

change filesystem to XFS 

format the drive

 

will that invalidate the parity and force it to be recomputed? 
 

very keen to avoid that if possible as it takes a few hours short of 2 days to complete the parity process 😬

 

for the sake of clarity ... do I need to remove all the existing content from the 18 TB drive ... say going into the disk share and deleting everything.

 

I could see that changing the parity to show no content for that disk

 

then changing the format to xfs ... and then copying the data back

 

Given it’s been pointed out already that the parity data effectively has knowledge of the format of the filesystem ... I’d just like to avoid triggering a parity recomp when I click format and change the file system ... even on an “empty” volume

Link to comment
4 minutes ago, belorion said:

will that invalidate the parity and force it to be recomputed? 

No, it remains valid.

 

5 minutes ago, belorion said:

for the sake of clarity ... do I need to remove all the existing content from the 18 TB drive ... say going into the disk share and deleting everything.

No, you just need to have it in another place since formatting will delete everything.

 

 

Link to comment
On 3/16/2021 at 1:21 AM, JorgeB said:

No, it remains valid.

 

No, you just need to have it in another place since formatting will delete everything.

 

 

Ok ... I tried that ... and I wasn't able to actually change the filesystem format - it was disabled.

Then I saw another post that said it was just a matter of stopping the array ... changing the format

Starting the array

 

and now it says my disk is unmountable .. and no mounted in the the size and used column of main page

 

https://forums.unraid.net/topic/39296-how-to-reformat-existing-drive/

 

Suggestions???

Link to comment

I see the PEBKAC now .. not used to the new UI .. there's an option displayed off the page (too many drives) which shows

unmountable disk present .. and gives option for FORMAT 🙂

 

To close the loop ..formatting fixed the issue.... and all my user shares returned once the "poison-pill" drive was formatted 🙂
Now to restore the content.

 

Thanks for your assistance and patience @JorgeB

Edited by belorion
  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.