August 12, 201411 yr Author and: ########### reiserfsck --rebuild-tree started at Tue Aug 12 03:49:47 2014 ########### Pass 0: ####### Pass 0 ####### Loading on-disk bitmap .. ok, 557275639 blocks marked used Skipping 30567 blocks (super block, journal, bitmaps) 557245072 blocks will be read 0%block 16983471: The number of items (768) is incorrect, should be (1) - corrected block 16983471: The free space (2048) is incorrect, should be (2000) - corrected pass0: vpf-10110: block 16983471, item (0): Unknown item type found [84150017 524288 0x8000701 (15)] - deleted left 538470489, 30677 /sec looks like this will take many hours...will post the results from Austin, later today/tonight.
August 12, 201411 yr Your plugins from looking at this latest syslog: Aug 11 11:51:16 unRAID logger: Installing system plugins Aug 11 11:51:16 unRAID logger: installing plugin: dynamix.plugin.control-2.0.3-noarch-bergware Aug 11 11:51:16 unRAID logger: file /tmp/plugin-startup: successfully wrote INLINE file contents Aug 11 11:51:16 unRAID logger: /bin/bash /tmp/plugin-startup ... missing Dynamix webGui plugin !!! Aug 11 11:51:16 unRAID logger: failed Aug 11 11:51:16 unRAID logger: installing plugin: dynamix.webGui-2.2.5-noarch-bergware This indicates that there is still a copy of dynamix.plugin.control in the system plugins folder /boot/plugins, and it is trying to install it first (because it is first alphabetically), but it fails because the main dynamix.webGui plugin has not been installed yet. Then later Aug 11 11:51:17 unRAID logger: success Aug 11 11:51:17 unRAID logger: file /var/log/plugins/dynamix.webGui: successfully wrote INLINE file contents Aug 11 11:51:17 unRAID logger: plugin successfully installed Aug 11 11:51:17 unRAID logger: Installing user plugins ...Aug 11 11:51:18 unRAID logger: installing plugin: dynamix.plugin.control-2.1.1-noarch-bergware Aug 11 11:51:18 unRAID logger: file /tmp/plugin-startup: successfully wrote INLINE file contents Aug 11 11:51:18 unRAID logger: /bin/bash /tmp/plugin-startup ... success so dynamix.plugin.control is installed a second time from user plugins in /boot/config/plugins and this time it succeeds because dynamix.webGui has already been installed. TL;DR: You still need to remove dynamix.plugin.control from /boot/plugins and just leave it in /boot/config/plugins where it belongs.
August 13, 201411 yr Author ok, i'll try to get the plugin situation cleaned up next, but after a grueling day of flying on 1hr of sleep, i finally got my computer set up here and was able to pull a copy of the reiserfsck log copied and zipped up (it's HUGE...took 15hrs or so, found what appear to be TONS of problems. since the log, even when zipped up, is still 2.9MB, i've had to share it via my Dropbox: https://www.dropbox.com/s/hkb56ajdgz3mjdv/reiserfsck_md13_2014-08-12.txt.zip in a nutshell, the final read-out says: Flushing..finished Objects without names 77048 Empty lost dirs removed 339 Dirs linked to /lost+found: 2943 Dirs without stat data found 86 Files linked to /lost+found 74105 Objects having used objectids: 74098 files fixed 71243 dirs fixed 2855 Pass 4 - finished done 584099, 25 /sec Deleted unreachable items 22054 Flushing..finished Syncing..finished ########### reiserfsck finished at Tue Aug 12 17:23:34 2014 ########### to me that sounds pretty bad, and i do wonder, of course, which 22054 files and 339 directories were deleted...maybe a more seasoned log whisperer can shed more light on it...how should i proceed? fix the plugin situation, reboot the server and keep on trucking, or does this warrant further action regarding the array?
August 13, 201411 yr Don't worry about the plugins for now. They are actually getting installed correctly eventually. Just a minor detail with that one that you can clean up sometime. I will let others with reiserfsck experience comment on how to proceed with that. Do you have backups of the problem files?
August 13, 201411 yr Author i haven't even had the time yet to inspect md13 in more detail to see what its contents are...i have backed up almost all of my important data (some 12TB) onto my Bitcasa Infinite Cloud Drive over the past 8 months, so there is a very good chance that they are on there...i'm too tired to get into it now, but will definitely do so when i've gotten some sleep. and yes, the plugins i'm not worried about...seems like i'm almost there, and frankly, to me they are more eye candy than essential functionality, so it's not something i *have* to have running right away. with that said, i found only that folder called "dynamix.plugin.control" with a readme file and nothing else in it, and removed it...i think it was a leftover from a failed install attempt...so now boot/plugins has nothing relating to dynamix in it...i attached yet another syslog from after i did this and rebooted...if i still don't have it right, shame on me, but maybe you can tell me...it's awfully big, so there must be some errors in there...i don't know. more soon, and thanks again for hanging in there with me! syslog-20140812-220045.txt.zip
August 13, 201411 yr There is always a potential problem in sorting out data loss after a file system corrupt - although in this case it does seem potentially quite severe. There is another option possible with reiserfsck which is the --scan-whole-disk option. This makes the --rebuild-tree option even slower as it reads every sector on the disk looking for items that look like files, but does recover more data. This might mean that the items currently deleted as 'unreachable' would be found - but it can also result in previously deleted files and file fragments to be added to the lost+found folder and need subsequent sorting out. Whether this is something one wants to consider doing depends on the importance of any potential lost data and the status of your backups. The unRAID User share facility makes it very easy to manage a set of disks during normal use, but it does mean that after a failure where data recovery was not perfect you may well not be sure what was on a particular failed disk. I have added a cron job on my system that runs periodically to list the folders/files on each physical drive and puts the resulting listing on the USB stick. That is just so in the case of an unrecoverable failure of a drive I can identify which files were on that disk that I need to retrieve from backups. I am hoping that something like this capability is one that Limetech might be considering as a standard feature in v6 at some point.
August 13, 201411 yr Author yeah, it does look like the hit i took is pretty severe indeed...what a bummer. it's difficult for me to tell, after the fact, just exactly *how* severe, but i am pretty sure that the data loss is close to 1TB...the lost+found folder on md13 contains over 100,000 file-fragments and folders with numbers for names, so i shudder a bit when thinking about what exactly went amiss here. fortunately, i *did* go through the tedious task of uploading all that is important to my Infinite Cloud drive these past 8 months, so that i am pretty sure that whatever was corrupted already was backed up, but since i am a bit obsessive about my data and the order that contains it, i am still looking at days if not weeks of time spent trying to sync data back to my server and making sure that it's all in sync in two locations. not sure about doing an even deeper scan...i mean, now that it has already put close 1TB into lost+found, what would i potentially gain from doing an even deeper scan? *more* lost+found fragments? i suppose it's better to know than not to, right? so i would run the same thing again, but with the --scan-whole-disk *instead* of the --rebuild-tree option? and this would probably take days then (during which i couldn't be using the server), right? oh boy...but thanks for the detailed post and suggestion!
August 13, 201411 yr not sure about doing an even deeper scan...i mean, now that it has already put close 1TB into lost+found, what would i potentially gain from doing an even deeper scan? *more* lost+found fragments? i suppose it's better to know than not to, right? so i would run the same thing again, but with the --scan-whole-disk *instead* of the --rebuild-tree option? and this would probably take days then (during which i couldn't be using the server), right? If you do go though this route then the --scan-whole-disk is used in addition to the --rebuild-tree option. As you surmised this takes even longer that rebuild-tree by itself. The only advantage is that it might recover some of the files that the first time it was indicated as being deleted because they were inaccessible. You have to decide if any gain is worth pain of sorting through the large lost+found folder that you seem to have. I seem to have been lucky when I tried it as I only ended up with a small number of files/folders in lost+found.
August 13, 201411 yr Author well, if there was even a single file in that lost+found folder that i could even define as something...they are all just numbered files, without extensions, and folders with numbers, 112,630 of them, adding up to an incredible 2.21EB as reported by Finder...i had no idea a 3TB drive could even hold 2.21EB...kinda boggles the mind no, i think i may just accept this data loss as yet another artifact of this ridiculously complex life i've created for myself, and try to find some time to devise a strategy for syncing back my backed up data to the live server...it's incredible that this is the first time in probably 10 years that i've seen unRAID lose data (outside of user-error), and unRAID isn't even really at fault here, but protected me from far greater damage. the remaining issue that does bother me very much, though, is the fact that i still cannot get my unRAID User Share or even the unRAID server as such, to mount in my Finder properly...i see the unRAID and unRAID-AFP shares, but when i click on them, they will say "Connection Failed"...if i click on the "Connect As..." button underneath, the login requestor won't come up either...until before all of this all happened, i could simply click on unRAID, then see all of my disk and user-shares, then click on the unRAID User Share to see all of my neatly organized data directories...i now have to go the Finder's "Go/Connect to Server..." menu to access all of my individual disk shares and the user share...why would they all show and work perfectly well through the menu, but not through the Finder? should i start a new topic for this, or might i get some help from you fine people?
Archived
This topic is now archived and is closed to further replies.