Rattus

Hello Again JorgeB and all, So its been a process but: - BTRFS Restore moved all of the files leftover files from cache to disk 1 under /mnt/disk1/restore - I've now got the pool back up, Pool is formatted to BTRFS and showing all available storage (see screenshot attached) I followed this method from Squid with some tweaks to reformat the pool. I had to remove the SATA Drive, reduce the pool size to one, then format as per Squid, then re-add the drive. Now the dev stats command shows no errors: My question now is, what is the safest way to move the files from the restore folder I created back to the drive so that they are recognised as never actually leaving the cache? Is that even what I am supposed to do here? Is it just the BTRFS Restore in reverse?

Hey JorgeB, Thanks heaps for the super quick replies yesturday. I've been trying to do the restore like you said but I've run into some problems. I've tried multiple commands, These are the errors I'm getting: So it won't work if the array is started, then it won't work if they array is stopped, and it also won't work if the array is in maintenance mode. I am super confused as to what I need to do here ...

No, Unfortunatly not. My Docker Image is still there along with some Nextcloud and swag data. How do I force it to move off? I have a spare sata SSD and sata port that I can use if that helps?

Hi All, I've run into a bit of an issue that I think is stemming from some corruption errors in my cache. Initial symptom was that the cache wasn't being emptied when mover was running. I followed this guide to get everything off of the cache: Backing up the Pool to the Array During the move, I thought I noticed that the pool still wasn't emptying. In fact it was emptying, it just wasn't giving me more free space. Please see attached 2 screen shots taken over a few hours. You'll note that I have 2 x 1 TB Drives in a BTRFS Raid 1. Initially 418GB was Used. Then 284 GB was Used but the Free Space only went up by about 10 GB. Both Drives are identical in size so no funny calculations on the BTRFS side. After some sleuthing I found this thread because I also got the same error message mentioned. BTRFS Pool Too Many Profiles. I've also now run the dev stats command and got this: I had some problems a while ago with bad RAM corrupting my cache, the sdc drive was added after this corruption was solved (or so I thought). So my question is: If I was to remove the NVME drive, zero it, and then re-add it, would that solve my issue? or is there more at play here that I haven't yet found? Do I need to do anything to be able to safely remove the NVME Drive? Results of the Pool Balance are below for reference (I'm not 100% sure how to tell if it is balanced and I can remove a drive) radon-diagnostics-20230419-1456.zip

Well Gents, Its been a rough ride, and a rough couple of days. But the "btrfs check --repair" did the trick, the disk mounted, i rebooted just to make sure everything reloaded ok, and all my data is back, my docker containers are working fine, and all is good with the world. Thanks heaps for your help, and answering my stupid questions, hopefully your answers can help someone else in a similar boat to me down the line. I think has reminded me how past due I am for an offsite backup of these config and other important files. Time to investigate a backblaze type of solution. Thanks again everyone who commented, much appreciated and much Love 💗 Rattus

Oh Lorde, I hope not ... I guess we will see. So that will be the nerve racking section, I'm going to try that tomorrow after I've had sleep and will hopefully be able to concentrate and not f it up. I'm guessing a few things final things here, could you please clarify/answer the below?: There's no way to compare what was copied vs what is on the drive to make sure it was all moved across before I nuke the cache disk? Restoring the data is as simple as copying the "restored" data to the freshly formatted cache disk? and unraid will know what to do with it? or will I need to put in the correct place on the array? If formatting the disk will completely erase it anyway, is there any harm in trying the "btrfs check --repair /dev/nvme0n1p1" command and seeing if that will fix it? and then if it doesn't work then do the format? assuming that point 3 is not recommended/doesn't work, I assume that xfs would be a better format for the cache disk? just based on reading through things with this whole experience. last question. so we are working on the assumption that the faulty ram module caused the corruption. if that problem happens again, and I have 2 cache drives in a parity type of arrangement, however unraid actually manages that, having them in party/replicated wouldn't help because the bad ram will affect both? doing that particular setup will only help with a drive failure? Well guys, if we can get answers to the above, hopefully the final post in this thread is that it's all up and running again. I want to say, what ever happens now, you guys have been great, thanks for speedy reply's, and hopefully I have good news for you tomorrow.

Quick update, I moved forward with the guide to see if any other errors might shine a light on what is happening. I didn't get any errors this time and now the following command is running and files are being copied to the destination: > btrfs restore -v /dev/nvme0n1p1 /mnt/disk4/restore there have been a few times when it has said something along the lines of: > we seem to be looping a lot on <file name>, do you want to continue? (y/n/a) It seems like it's only been doing that on rather large files (databases, or logs, and now my vm's), I can live without the logs, and probably the vm's, but will the database files be ok? Also, it is clear that the files are still there and they are (relatively), ok, how do I go about bringing the cache disk back? hopefully without nuking it? (fingers crossed)

So I'm confused, what am I doing wrong, the guide says use mkdir command, this what I have just done with the output and it's giving me a very similar output. The directory is there according to the LS command. From the guide:

Thanks again for the help thus far gents. Ok Update, I have put memtest on its own USB and did the test on the server. It was showing errors. Lots of errors. Memory setup was 4x8GB DDR4 modules. Removed one pair of DIMM's ran test. Test Ok. Swapped DIMM pairs, Errors. Removed 1 stick, no errors. Swapped with other stick pair, errors. Moved erroring DIMM to different slots to ensure wasn't bad traces on mobo still errors in multiple slots. Conclusion 1 bad stick. Stick removed from system and quarantined. Concurrently ran memtest on gaming rig. gaming rig showed no errors. Memory setup on rig was 4x16GB DDR4. Gaming rig ram has now been moved to Server, Ran Memtest (for safety) No errors. Now that I have known good ram in the server how should I proceed in recovering the cache?

Ok ... Bit of an update. Found Memtest, Rebooted to memtest, server went into kind of boot loop where it bought up the blue boot menu, it auto selected memtest, attempted to run, and rebooted, then did the same thing over and over. Assuming it was a ram issue, I removed the ram and replaced it with assumed good ram from my gaming rig. same thing happened. original ram is now back in the server. Note, server would boot into normal unraid with both sets of ram, but neither set would do memtest. Clearly there is something wrong with the system overall, so I will look at replacing it, but in the mean time is there any way of recovering the data on cache drive? my appdata folder was apparently on there and not on the main array so I've got none of my setup data that I would need if I was to rebuild the server with new hardware. On a side note, I have ordered to 2.5" ssd's that I will use for the cache so that there is redundancy in the future, but if this is a ram issue then that actually wouldn't make a difference would it? Thanks for the help thus far @JorgeB

ok, this going to sound stupid ... how does one run memtest on unraid. my googlefu is failing me tonight ...

Sorry, that was my bad in editing, detail is now where it should be and diagnostics attached to this post radon-diagnostics-20220330-0054.zip

Thanks for that guys, sorry to be so nieve, i guess i was just panicking. I've started a new issue so that I don't clutter this one. Any chance of getting some help? I've put in the post the terminal commands and outputs. Thanks again guys, last post here.

Hey guys, So i seem to have run into the same issue as a few people, and specifically this post, I have been trying to work through the FAQ but I'm not having any luck. I'll copy the terminal commands and output below. From the linked FAQ post instructions: From BTRFS File system check advice on unraid: Different variation of the same BTRFS guide: From the Log: Does anyone think they might be able to point me in the right direction on this one? Thanks, Rattus

Hey guys, So I've just run into the same problem as you, I know this was a while ago for you but if you can remember what you did could you please share? I was hoping that you linked to the FAQ you mentioned but it's not there. Thanks heaps!

Hi Guys, I was hoping that someone could help me, my JellyFin instance recently decided it needed an update so I did and now it refuses to start. All of my other containers start no issue. The error it gives is: I have checked the logs and there is nothing there at all so there is no other indication as to why the failure. I was looking for others with the same issue and there seems to be some. I found this thread with the exact same issue but there doesn't seem to be an answer that works here. Can someone please assist in sorting this issue? https://forums.unraid.net/topic/117030-jellyfin-execution-error-server-error/