FryGuy
-
Posts
23 -
Joined
-
Last visited
Content Type
Profiles
Forums
Downloads
Store
Gallery
Bug Reports
Documentation
Landing
Posts posted by FryGuy
-
-
Disk 2 FS block uncorrectable. I'll have the new disks hopefully next wednesday or thursday to move data over to XFS and get recovery underway.
-
Disk 2 had 1 FS error block, running a scrub repair now. Disk 3 had no FS errors. I got no reported errors on I/O.
-
1 hour ago, johnnie.black said:
See here on how to reset current error count and monitor for new ones (you'll need to adjust the path), then post new diags if/when there are more errors (before rebooting).
Set the script to run hourly like described above, reset the error count, now running scrub on the two BTRFS disks. I'll keep and eye out for errors, fingers crossed.
-
1
-
-
20 hours ago, FryGuy said:
No its not good. I missed that. Shutting down now and doing a memtest.
Ran two cycles of the latest version of memtest, just to be sure. Nearly 20 hrs testing on the 32GB ram and returned no errors. Any other ideas? I hope you don't say SATA controller, its built on the mobo. 😬
-
3 minutes ago, johnnie.black said:
I'm sorry, so what is the problem exactly? If you want to attempt to recover data form am unmountable btrfs disk I need diags after trying to mount it.
What is crashing, the server? I though this was about an unmountable disk, again I'm sorry I didn't read the entire thread, so please do a quick summary if there are other issues.
Also this is not a good sign:
Aug 3 13:26:08 unRAID kernel: BTRFS info (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 ... Aug 3 13:26:21 unRAID kernel: BTRFS info (device md3): bdev /dev/md3 errs: wr 0, rd 0, flush 0, corrupt 435370, gen 0
Both disks 2 and 3 are showing corruption errors, this is usually data corruption caused by a hardware problem, like bad RAM, it would be a good idea to run memtest.
No its not good. I missed that. Shutting down now and doing a memtest.
-
2 minutes ago, johnnie.black said:
To see in what state the filesystem is I need diags after attempting to mount that disk.
The other disks are mounting, I just don't know why it's crashing still. I just started it up 1 hour and 40 mins ago to start the scrub if that's fresh enough for the diags. Here is is.
-
7 hours ago, johnnie.black said:
I dint' read the complete thread but I see disk1 is now formatted xfs, so you won't be able to use any btrfs recovery tools, you might still be able to get something with UFS explorer.
It's another disk I have in there now. The corrupt drive is out in a USB dock for recovery. I have more disks arriving for future backup. I'm going to use to move the array to xfs disks. The damaged file system disk is only disaster recovery now, worse case I re-rip source material as I find out what has been lost. I may be able to at least use UFS Explorer to find out was has been lost, as it is listing all the file names and sizes. I'm hopeful that it will recover the files too but it don't appear to preserve the dir structure, oh well. Critical stuff I have backups of on disks and cloud. I now think I have errors on my other disks causing the system to freeze during parity check. I run a scrub last night to check the disks and forgot to turn off the scheduled weekly parity check and that froze the system up. Now running scrub again with parity check schedule disabled, by morning i'll know if it worked.
-
10 hours ago, johnnie.black said:
Please post the diagnostics: Tools -> Diagnostics
-
1 minute ago, trurl said:
I don't see a missing disk in that screenshot, nor do I see a disabled disk. A missing or disabled disk would not be green, it would have a red X instead.
So, parity is not emulating anything. You just have an unmountable disk, which usually means a corrupted filesystem.
Ok that confirms my worst fears, hard to visually check and confirm every file/dir with such a large pool. That's exactly why I didn't want to start anything up and mess up my cache and database. Looks like it's going to be disaster recovery after all. I tested with UFS Explorer it looks like it will recover everything. Do you know of any other free BTRFS recovery tools (or i'll just web search)? It doesn't preserve dir structure but at least i'll get my files back from that corrupt backup. Thank you, terminology confusion and all.
-
4 minutes ago, trurl said:
This is a bit confusing and makes me question your use of "emulated" again. Maybe a screenshot would clarify.
I don't know what else you would call it when the array is running with the parity "emulating" the missing disk as part of the array. Clear english definition usage of the word to me.
-
4 minutes ago, trurl said:Quote
For future reference, rebuilding to the same disk is simple
- Stop array
- Unassigned disk to be rebuilt
- Start array with disk unassigned
- Stop array
- Reassign disk to be rebuilt
- Start array to begin rebuilt
Other than the un-necessary step of clearing (format first then preclear next try) the disk this is what i had did.
4 minutes ago, trurl said:Not sure what happened to your filesystem. Perhaps if you had asked for help sooner someone would have had some better idea how to recover. I have seen mention that btrfs recovery tools are perhaps not as well developed as XFS, for example. I personally don't use btrfs in the parity array, but I know @johnnie.black has worked with it.
As mentioned, rebuild seldom fixes filesystem corruption. Since parity updates happen at a very low level, parity usually will be in sync with the low level bits, even if they are a corrupt filesystem. One thing you can try if you find yourself in a similar situation is to start the array with the problem disk unassigned. That will make Unraid emulate the disk from parity, and whatever the result of that emulation is will be the result of the rebuild. So if the emulated disk is still corrupt, rebuild will not help.
The array will start with the emulated disk and everything is green. I haven't dared to try VMs or Dockers to see if everything is there to risk the disk writes. Without running some things like Plex i'm not sure if anything missing.
4 minutes ago, trurl said:QuoteOften in situations where people actually have a disabled disk, and the emulated disk is corrupt, we will have them repair the filesystem of the emulated disk before actually doing the rebuild.
The disk isn't showing as disabled, its marked as green.
4 minutes ago, trurl said:I would have asked for diagnostics at the very beginning so I might have had a clearer idea about your configuration and situation, but it seemed like it was maybe too late for any advice I might have provided.
The original issue with the server that was causing the kernel panic problem was solved with updates. I don't think i have the logs of the issue anymore as it was resolved. Ran scrub / repair on disks but by times updates fixed the issue the damage was done. It should have been a trivial fix to just backup rebuild a drive. Turned out to escalate into a situation.
Now how do i figure out if my emulate disk is corrupt? I'll get the two more disks (assuming I overwrite the corrupt backup I got of the emulated disk, would rather not but $$$) to pull the whole array data off, i should have a current backup set on the shelf anyway. This is the only way I see to proceed forward with recovery if the emulated disk is good.
-
1 hour ago, trurl said:Quote
What do you mean by the data that is emulated?
If you have a missing or disabled data disk, and all other disks including parity are good and parity is in sync, then the data for the missing or disabled disk is emulated from the parity calculation by reading parity plus all other disks. If you have single parity then a single disk can be emualted, if dual parity then 2 disks can be emulated.
But your use of the word emulated seems a bit more vague that all that.
If you read my post with my disk layout you wouldn't be debating the semantics of what emulated means. I have one parity disk with three data, all 14 TB.
QuoteQuoteEach data disk in Unraid is an independent filesystem, so each individual disk can be read all by itself on any linux, including Unraid. Unless you think there is a problem with each and every one of your disks, then I don't see any point to backing up the whole array, and if there is a problem with each and every one of your disks, then backing them up may not be possible.
Every other disk is fine just can't rebuild from the parity but it can start array and emulate the missing disk. I don't understand how or why this happened that's why i'm here. Originally i was thinking that i did something horribly wrong in the procedure but now i'm reassured that i didn't. The only thing i don't understand is how my new drive backup became corrupt with cp -r command too? I'm sure i didn't get the drives mixed up with one on a USB bus.
QuoteYou should have backups of anything important and irreplaceable, of course, but you indicated you did.
QuoteReally everything you say you did seemed to be wrong or pointless. For example, as mentioned, when rebuilding a disk, the entire disk is going to be overwritten anyway. So preclearing the disk and / or formatting the disk actually is completely pointless.
I think this is the point of confusion. It was the original disk 1 that had the corrupt BTRFS it was still R/W with 5 unrecoverable errors, it stayed in the same slot the new disk was used to backup its data. I perhaps wrongly thought it was safer if a rebuild went wrong,
At the time i didn't know how else to get the original disk to appear as a new disk to rebuild.
QuoteMaybe rebuilding was pointless too. If you had a corrupt filesystem, then rebuilding isn't going to help.
If a corrupt FS can affect the parity to cause it to not rebuild then i'm *expletive deleted* either way. Nothing I could have done from the start would have solved the problem. I'm an experienced user of linux and seldomly reach out for help, I usually find it so condescending and toxic on forums. This is a sh*t hit the fan moment and i'm not sure what even happened myself.
-
19 minutes ago, trurl said:Quote
Wish you had asked for advice earlier, probably would have been better if you had asked before doing anything at all.
I really thought it was going to be a straightforward, remove from array, reformat, re-add, start, and allow to rebuild kind of deal.
I
Quoteit's not entirely clear from your description what you had or what you did.
I backed up data to New Disk from the corrupt BTRFS Disk 1 in the server, Disk 1 remove from confg, shutdown, format Disk 1 in unassigned devices, add to array, start array, it didn't say building disk, immediately shutdown array, maybe i got a panic that something was wrong and nothing was. Then ran a preclear on Disk 1, started array and proceed to build drive, drive rebuild failed with "Unmountable: No file system" repeated twice with the same results.
QuoteA mistake some make is formatting a disk in the array, then expecting parity to rebuild the data that was there before the format. Since parity is updated by the format (and any other write operation in the array) rebuild will just result in a formatted disk.
But that isn't what you say you did. Formatting or clearing a disk that is not in the array will not affect parity, but it is pointless since rebuild is just going to completely overwrite it anyway. Rebuilding in that situation would simply result in the same data that was on the disk before. Probably you already had a corrupt filesystem, and so a corrupt filesystem is the expected result of the rebuild.
Then there should be no way that what i did corrupted the parity, the parity was valid before with no errors.
QuoteOn the other hand, you don't explicitly mention rebuilding anything (except in the title). Instead you say
Did you ever do New Config during any of this?
I used the wording added it back, it was rebuilding the drive. Sorry for the vague description above, i was trying to keep it short it already seemed so long winded. No new config and i didnt use the format the drive check box either, i know that will definitely mess things up if i did that. New config, especially if you get the drive order wrong if i'm not mistaken.
QuoteMaybe UFS Explorer will help, I've never used it.
It does show all the files but it wont preserve any dir structure. Its my last resort if I must.
I guess the is no way to backup just the data that is emulated? It don't show up as /mnt/disk1 as its not a mounted disk is there a location that the virtual disk mounts to for the pool?
If not then I need to do a full tar backup of the whole array to external drives to be sure i have everything on the array?
-
I originally had Disk 1 with BTRFS uncorrectable errors from kernel panics and forced shutdowns. Found and fixed the offending software with updates but the damage was done. (One parity, Three data, 14TB WD DC HC530's) Had got a extra drive to backup the data to (foreshadowing here) first. Backed up the data from command line, error on some temp and couple corrupt files of no value. Confirmed the files on the drive with a ls in a few main dir's, all is good.
Removed the drive from the array config, shut down, now my mistake (I know RTFM, I should have read it again), formatted the drive in unassigned devices, added back to the array and hit start, had a oh *explitivitive deleted* moment, shut it down. Precleared the drive, then added again. Think in hindsight the damage was already done. That's why I made a back up, drive won't mount! I'm not sure how but in the copy process (cp -r) to the brand new drive it also corrupted the BTRFS on that drive also. Now the original source drive is gone and the backup is corrupt, nice! It's damaged beyond BTRFS repair tools capability but recovery looks possible as last resort with UFS Explorer but it looks like it will lose all the dir structure. It looks to be still emulating that disk, any way i can pull just the data of that emulated disk? At this point it almost looks like i got to bite the bullet and buy two 14TB hard drives! I do have seven 4TB hard drives with an older backup on it. The new data set has outgrown the disks size, even then I don't feel comfortable overwriting it with everything else that's happened. I got archival and cloud backup of everything really critical.
I guess my question is what are my options moving forward for data recovery at this point? Thank you.
-
6 hours ago, itimpi said:
If you want any such change to survive a reboot then you need to make a copy of the file(s) on the flash drive and add an entry to the ‘config/go’ file on the flash drive to copy it into position (and potentially set correct permissions) as part of the system start up process.
Since Unraid does not support traditional users in the Linux sense not sure what you have done to allow you to log in with a non-root user.
I worded that wrong, non-root user as in not root user name, however. I used the "Users" under "Settings" and the user isnt part of the root or adm when I check groups, so thats non-root?
config/go file I'll try that out. thnak you.
First time poking around under the hood. Is there any detailed system documentaion? (not user documentation)
-
I added the .bash* files in the /root with the bash_aliases, .bash_exports, .bash_profile, .bash_wrappers, .bashrc files from https://github.com/flipsidecreations/dotfiles with my changes. This is working now on first login with my non-root user.
Not the behaviour i expected. I figured that bash would get its config from the logged in users home dir.
Not sure if this will surrive a reboot yet.
-
Looking at customizing bash colors. I'm not sure how to get it to stick. I have a .bashrc with;
PS1="\[\033[0;31m\]\342\224\214\342\224\200\$([[ \$? != 0 ]] && echo \"[\[\033[0;31m\]\342\234\227\[\033[0;37m\]]\342\224\200\")[$(if [[ ${EUID} == 0 ]]; then echo '\[\033[01;31m\]root\[\033[01;33m\]@\[\033[01;96m\]\h'; else echo '\[\033[0;39m\]\u\[\033[01;33m\]@\[\033[01;96m\]\h'; fi)\[\033[0;31m\]]\342\224\200[\[\033[0;32m\]\w\[\033[0;31m\]]\n\[\033[0;31m\]\342\224\224\342\224\200\342\224\200\342\225\274 \[\033[0m\]\[\e[01;33m\]\\$\[\e[0m\]"This is from https://github.com/flipsidecreations/dotfiles I just love the look of this.
This works if i then type "bash" after login but not at login. Not sure if it will survive a boot being in the home dir.
Any guidance will be apreciated.
-
Using the Ripper docker for the first time in a long time and now getting this error. Deleted the Docker and appdata/settings and recreated and still getting this error.
cp: cannot create regular file '/root/.MakeMKV/': Not a directory
Any ideas what i screwed up?
-
8 hours ago, peter_sm said:
OK, Try this ,
Install easyrsa again.
edit easyrsa, on line 900 change ...
original
-out "$pkcs_out" "$pkcs_opts" || die "\
edit ( remow the quotation marks"
-out "$pkcs_out" $pkcs_opts || die "\
But preferred is a inline file.
//Peter
Fantastic, That fixed it! Thank you.
-
2 hours ago, FryGuy said:
Thank you. I'll give it a try.
Gave it a shot. I must be putting the files in a wrong location or something. /mnt/user/appdata/myVPNserver or cache location and replaced the current easy-rsa folder but unable to generate certs and keys and tells me easy rsa not downloaded. even downloaded the latest easy-rsa from interface and then replaced all the similar named files with the v3.0.4 ones still the same. what the hell am i doing wrong?
-
13 hours ago, peter_sm said:
Looks like they have got bug in easyrsa 3.0.5. Branch 3.0.4 works fine.
Try to 3.0.4 and test. unzip and rename to easy-rsa in your config folder
Thank you. I'll give it a try.
-
Not creating the .p12 file for some reason. Any ideas why?
Log output below
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Adding client: USER
spawn ./easyrsa build-client-full USER nopass
Generating a 2048 bit RSA private key
.........+++
................................................................+++
writing new private key to '/mnt/cache/appdata/myVPNserver/easy-rsa/easyrsa3/pki/private/USER.key.XXXXwylxGo'
-----
Using configuration from ./openssl-easyrsa.cnf
Enter pass phrase for /mnt/cache/appdata/myVPNserver/easy-rsa/easyrsa3/pki/private/ca.key:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
commonName :ASN.1 12:'USER'
Certificate is to be certified until Feb 2 01:32:01 2028 GMT (3650 days)Write out database with 1 new entries
Data Base Updated
spawn ./easyrsa export-p12 USER
Usage: pkcs12 [options]
where options are
-export output PKCS12 file
-chain add certificate chain
-inkey file private key if not infile
-certfile f add all certs in f
-CApath arg - PEM format directory of CA's
-CAfile arg - PEM format file of CA's
-name "name" use name as friendly name
-caname "nm" use nm as CA friendly name (can be used more than once).
-in infile input filename
-out outfile output filename
-noout don't output anything, just verify.
-nomacver don't verify MAC.
-nocerts don't output certificates.
-clcerts only output client certificates.
-cacerts only output CA certificates.
-nokeys don't output private keys.
-info give info about PKCS#12 structure.
-des encrypt private keys with DES
-des3 encrypt private keys with triple DES (default)
-seed encrypt private keys with seed
-aes128, -aes192, -aes256
encrypt PEM output with cbc aes
-camellia128, -camellia192, -camellia256
encrypt PEM output with cbc camellia
-nodes don't encrypt private keys
-noiter don't use encryption iteration
-nomaciter don't use MAC iteration
-maciter use MAC iteration
-nomac don't generate MAC
-twopass separate MAC, encryption passwords
-descert encrypt PKCS#12 certificates with triple DES (default RC2-40)
-certpbe alg specify certificate PBE algorithm (default RC2-40)
-keypbe alg specify private key PBE algorithm (default 3DES)
-macalg alg digest algorithm used in MAC (default SHA1)
-keyex set MS key exchange type
-keysig set MS key signature type
-password p set import/export password source
-passin p input file pass phrase source
-passout p output file pass phrase source
-engine e use engine e, possibly a hardware device.
-rand file:file:...
load the file (or the files in the directory) into
the random number generator
-CSP name Microsoft CSP name
-LMK Add local machine keyset attribute to private keyEasy-RSA error:
Export of p12 failed: see above for related openssl errors.
send: spawn id exp5 not open
while executing
"send "PASSWORD\r""
cp: cannot stat '/mnt/cache/appdata/myVPNserver/easy-rsa/easyrsa3/pki/private/USER.p12': No such file or directory
Update USER.ovpn to be used with IOS
Creating a zip file for the client
zip warning: name not matched: USER.p12
adding: USER.ovpn (deflated 33%)
adding: README.txt (deflated 53%)
Client files have been stored in this folder ..
/mnt/cache/appdata/myVPNserver/clients/USER
(6.9.0-beta25) Corrupt BTRFS, rebuild now "Unmountable: No file system"
in General Support
Posted
No I/O errors. Found that Disk 2 has 1 unrecoverable FS error. Just got my new drives, time for recovery.