DaveW42

I was particularly attending to that last line in the second output, when I put forward the imaging idea. "Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this." We're not quite there yet (haven't tried mounting the drive), but I am trying to avoid the possibility of corruption and thought it might be prudent to consider imaging. Dave

Thanks, JorgeB. So far I've just run xfs_repair twice, both times using the GUI. The second time the output indicated I would need to mount the drive so it could replay (?) the log. That was the next step, but before I did that I thought about imaging the drive (and wrote the corresponding post above). Per your request, below is all of the output I have seen from xfs_repair. Dave The first time I ran it with the -nv option and output was as follows: Phase 1 - find and verify superblock... bad primary superblock - bad CRC in superblock !!! attempting to find secondary superblock... found candidate secondary superblock... verified secondary superblock... would write modified primary superblock Primary superblock would have been modified. Cannot proceed further in no_modify mode. Exiting now. The second time I ran it with the -v option and output was as follows: Phase 1 - find and verify superblock... bad primary superblock - bad CRC in superblock !!! attempting to find secondary superblock... .found candidate secondary superblock... verified secondary superblock... writing modified primary superblock - block cache size set to 299848 entries sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128 resetting superblock root inode pointer to 128 sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129 resetting superblock realtime bitmap inode pointer to 129 sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130 resetting superblock realtime summary inode pointer to 130 Phase 2 - using internal log - zero log... zero_log: head block 1075093 tail block 1075089 ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this.

Hi, JorgeB. Below is the text on the wiki that I was referring to (I've changed the text color to green for the parts I was referencing). Thanks, Dave Additional comments The xfs_repair tool may take a long time on a full file system (several minutes to a half hour, or more). If the xfs_repair command fails, and we're hearing numerous reports of this(!), then you will have no recourse but to redo the drive. Use the instructions in the Redoing a drive formatted with XFS section below. We're sorry, we hope there will be better XFS repair tools some day! If there was significant corruption, then it is possible that some files were not completely recovered. Check for a lost+found folder on this drive, which may contain fragments of the unrecoverable files. It is up to you to examine these and determine what files they are from, and act accordingly. Hopefully, you have another copy of each file. When you are finished examining them and saving what you can, then delete the fragments and remove the lost+found folder. Dealing with this folder does not have to be done immediately. This is similar to running chkdsk or scandisk within Windows, and finding lost clusters, and dealing with files named File0000.chk or similar. You may find one user's story very helpful, plus his later tale of the problems of sifting through the recovered files. If you get an error indicating something like trouble opening the file system, it may indicate that you attempted to run the file system check on the wrong device name. For almost all repairs, you would use /dev/md1, /dev/md2, /dev/md3, /dev/md4, etc. If operating on the cache drive (which is not protected by parity), you would use /dev/sdX1 (note the trailing "1" indicating the first partition on the cache drive). If you want to test and repair a non-array drive, you would use the drive's partition symbol (e.g. sdc1, sdj1, sdx1, etc), not the array device symbol (e.g. md1, md13, etc). So the device name would be something like /dev/sdj1, /dev/sdx1, etc. Redoing a drive formatted with XFS If you are here because the XFS repair tool has failed you(!), then the best we can recommend is to save the data, reformat, and restore the data (NOT a desirable course of action)! Do your best to copy off everything you can, to a safe place. If something important is absolutely needed and still inaccessible, try File Scavenger or a live CD of TestDisk. Change the file system format for the drive to ReiserFS (just to reset the formatting, it's temporary and fairly quick) Start the array and format the drive Stop the array Change the file system format for the drive to XFS again Start the array and format the drive again Copy back everything you saved It's certainly not a welcome method, but it does produce a fresh and clean XFS format. (The write-up above has not been tested by this author. If corrections are needed, please do them, or PM RobJ with the corrections or suggestions.)

Hi, JorgeB. I'm pretty nervous about possibly losing data (family photos, etc.), especially since I can't precisely even say what data was on that drive. Correct me if I am wrong, but I believe UNRAID can choose to write to different hard disks as it writes files to a single share, and thus--if I lose files on this drive--there are likely to be shares where one or more files have gone missing while other files (housed on different disks) are still present. It is bad enough to possibly lose files, but finding holes in my data would take things a step further and drive me crazy! Would it be a good idea to create an image of the drive before doing anything that might change data on the drive? Then I could restore the image if the repair encounters problems, and restart recovery efforts using the tools I mentioned above, or perhaps even other tools. I just ordered a couple more hard drives in case this might be helpful. I really want to avoid loss of data. Believe it or not, before all this happened I'd been trying to figure out the best way to backup critical files to an old synology server that I have. I thought with the parity protection I would be safe for a few months while I tried to figure this out. Live and learn! Also, below is what UNRAID said when I ran xfs_repair (I added "-v", so we get more output). Thanks, Dave Phase 1 - find and verify superblock... bad primary superblock - bad CRC in superblock !!! attempting to find secondary superblock... .found candidate secondary superblock... verified secondary superblock... writing modified primary superblock - block cache size set to 299848 entries sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128 resetting superblock root inode pointer to 128 sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129 resetting superblock realtime bitmap inode pointer to 129 sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130 resetting superblock realtime summary inode pointer to 130 Phase 2 - using internal log - zero log... zero_log: head block 1075093 tail block 1075089 ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this.

Thank you, JorgeB. I note that in the wiki you originally linked to (https://wiki.unraid.net/Check_Disk_Filesystems#Checking_and_fixing_drives_in_the_webGui), that unfortunately xfs_repair often doesn't work. Before I try this, would you recommend that I try looking at the drive through File Scavenger or a live CD of TestDisk? Those two options are noted in the wiki when you are redoing a drive formatted with XFS (i.e., after xfs_repair initially ails). I am definitely new to this, and am just wondering if it might be better try those first if, for example, there was a non-destructive mode in either of those pieces of software? I will definitely defer to what you recommend, I just wanted to throw this thought out there. Thanks again! Dave

Hi, JorgeB. I followed the instructions provided and ran xfs_repair on both disks using the GUI, with options specified as follows: -nv Both disks are showing the same message (see below). What should I do next? Should I run xfs_repair with no options? Thanks! Dave SYSTEM MESSAGE Phase 1 - find and verify superblock... bad primary superblock - bad CRC in superblock !!! attempting to find secondary superblock... found candidate secondary superblock... verified secondary superblock... would write modified primary superblock Primary superblock would have been modified. Cannot proceed further in no_modify mode. Exiting now.

Thank you, Jorge! I will follow those procedures tonight, after work. The drives are formatted XFS. Dave

Thanks for updating and posting the file! Dave

Help would be greatly appreciated. Over the past 2 months I've been running into hard drive problems with my UNRAID server. The server has 2 parity drives, 16 drives on the array, roughly two or three in unassigned devices, an NVME cache drive, an SSD cache drive, and a couple of separate drives outside of the array. The problem started when I saw 2 drives on the array with their status marked as disabled. I bought two new drives ASAP, followed UNRAID's standard procedures for replacing disabled drives (https://wiki.unraid.net/Manual/Storage_Management#Replacing_failed.2Fdisabled_disk.28s.29) and things looked ok for a short period of time. Then another drive registered as disabled. I looked to see if there was some commonality in the failures, and thought at first that the common thread was that the disabled drives were those attached directly to the SATA ports on my motherboard (Asus ROG Strix X570-E Gaming running AMD 5950x). I had had other issues with that X570-E motherboard in the past (i.e., it NEVER would accept more than 64GB of memory, even after two replacements boards from Asus), so I thought I would take the hint and move to an entirely different ASUS model of motherboard within the x570 family (ASUS ROG Crosshair VIII Formula AMD X570). Unfortunately that motherboard has the same memory issues as my X570-E Gaming, and shortly thereafter ANOTHER hard drive went disabled. I then logged the Serial Numbers and locations of all my drives and their physical locations in my server case, and saw that the commonality was actually attachment to one of two LSI SAS9211-8I 8PORT Internal 6GB Sata+SAS Pcie 2.0 cards. I had a spare LSI card, so I removed the bad one and replaced it with the spare. Everything seemed to boot up fine, and it seemed I could access files normally (although I still had 2 disabled drives). At some point shortly thereafter (30 min or an hour?) I noticed that the new LSI card had RAID firmware installed, instead of the IT firmware that you are supposed to use when you are running Just a Bunch of DIsks (JBOD). I then upgraded the LSI firmware on the new card, as well as the firmware on the original card (i.e., the one that always worked and continues to work). Both cards were updated to the recommended (v20) version of the IT firmware. I thought this would address the core hardware issue, and maybe it did -- but unfortunately I'm not quite sure yet (!) My next step was to rebuild the two disabled drives. I took one of the 10TB drives that was in the server but not yet in use (not part of the array, no data stored on it), and went to rebuild the smaller of the two drives (10TB) using the aforementioned standard procedures for replacing disabled drives. Yesterday the rebuild process completed, and things started to look ok. But then today I saw that that newly synced 10TB drive is appearing as "Unmountable: Wrong or no file system," as does the other drive that had also been disabled (a 14TB drive). The newly synced drive still appears as "green" (i.e., no warning triangle), and the short SMART test I ran on that drive seemed to indicate it what was doing fine. Up until this point I haven't seen any loss of data, across all of the issues indicated above. However, about an hour ago I went to look for a set of files and they were gone. This last part is more concerning to me than anything prior. I have now (via the GUI) removed the 10TB drive that was registered as unmountable from the array, and that disk (Disk 12) is now listed as not installed. The remaining drive that was disabled (14TB) also now appears as "Unmountable: wrong or no file system." Both drives have a red "X" next to them. Currently I have the array running, with those two drives set as indicated in the last paragraph. I have attached the diagnostic file for my system. Any help would be greatly, greatly appreciated in getting this system going again. Also, based on what you can see, do you think I lost any files? A few quick last notes as well. - With regard to the memory issue, I have sent the four 32GB sticks back for replacement, so right now I am just using an older 8GB stick for the server. - All dockers and VMs are shut down now. - In terms of changes/new things, the NVME is also new, and I recently attached the following unit to the system, to be able to run preclear etc. separately on new drives. (https://www.amazon.com/gp/product/B0759567JT/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1https://www.amazon.com/gp/product/B0759567JT/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1). Thanks again so much for the help! Dave nas24-diagnostics-20230212-1544.zip

Thanks, ChatNoir! I'm still on 6.8.2, so I'll need to wait on exploring that one until I have time to make sure I can upgrade to 6.9 safely. In short, I see from a spaceinvaderone video that docker container updates are recommended when I make the upgrade. I will want to make sure that--when I originally created those containers--I didn't inadvertently violate any of the guidance you and trurl provided above. Just want to make extra sure I don't lose anything. Thanks again! Dave

Wow, thanks so much ChatNoir and trurl!!! I've read and re-read both of your posts several times, and things are sinking in a bit more each time. ChatNoir, I liked your use of color to denote the Unraid vs. Docker side of things. That helps quite a bit. Interesting that the Linux file structure appears on both sides (on Unraid side and then in Docker nested within UnRaid). trurl, your use of the term 'view' in describing shares definitely provided an epiphany of sorts, and the clarification concerning disks and shares and the extra directories created by Unassigned Devices was extremely helpful. So, in essence we have an underlying file structure, and one can look at that structure through multiple perspectives such as Unraid's attached physical disks ("disks"), particular directories the user has chosen to make available ("shares") under particular constrains (i.e., permissions and configurations such as "private" and "public"), directories located on other computers that we want easy access to ("remotes"), and any "top level" folders on the disk. If I am reading this correctly--based on ChatNoir's comment concerning "/mnt within the container"--if I am navigating through directories in Unraid (i.e., not a docker), I should be able to find the files I create within "/mnt", with "/mnt/user" being particularly important (thanks trurl!). If I am in a Docker container, we only see what is directly inside the container and what we have specifically configured to pass through the container. Thus looking at "/mnt" is essentially useless, since I will likely want any file I create to go into one of the Unraid paths that I have mapped to appear in the container (i.e, so the docker doesn't get filled up, and also so I don't lose a file I have created if I later need to rebuild the docker image). Putting stuff in "/mnt" in a docker container would be a quick way to fill up the docker image, and a quick way to lose files if something goes wrong with the container. Thanks so much!!! This really explains why it always felt like trial and error to find a file I was interested in. In short, views on files are constrained when looking through the lens of a docker container. On the other hand, when you are looking at the Unraid file structure, there can be many ways to access the same file, depending on which "view" you choose to pursue. That is totally cool. But also a bit confusing! I think I am starting to put things together a bit. I'll have to look further into the concept of a "pool." I haven't seen that one before and will need to look into it to understand. Thanks again, and also sorry it took me a few days to respond (life called ...) Dave

Thanks, ChatNoir! I had read portions of that thread before ... each time I reread these things one more piece of information sticks. Within the link you shared, the post i have referenced below is particularly informative. To provide an additional example of where I struggle, consider within the post below the reference to directories such as "/mnt/user" etc. Where do I actually see these directories in the Unraid GUI? I can gain access to them if I open a terminal and navigate to them via the command line (e.g., "cd /mnt"), but are they accessible somewhere else too? "/mnt" seems to be a pretty darn important directory (I see elsewhere that the disks are all 'mounted' to this directory). And if I go into a file manager program like CloudCommander, for example, I can navigate to "/mnt" but then no subdirectories appear (i.e., no "/mnt/user"). Why is this the case? Is "/mnt/user" a hidden directory? The perceived inconsistencies here make it harder to figure things out and identify the basic principles and rules by which the Unraid file system operates. Having said all this, I really really do like Unraid quite a bit. I just wish I know more about what was going on, so I could better troubleshoot any issues as they occur. Also, I think Squid is absolutely incredible in terms of his contributions to this community (not trying to criticize the FAQ in any way, just trying to share where my ignorance lies so I can become more knowledgeable).

Thanks for your insight, trurl! I was aware of the case-sensitive nature of linux, but just made a mistake at the time. Actually, I wonder if the container template even forced that particular directory ("music") to appear ... for example, I see a typo in the Roon docker that labels "Backup" as "Backip" ... I've tried to just remove the "Backip" from the template when creating the docker container, but it keeps reappearing even after I delete it. Also, with regard to paths, can I just create whatever paths in a docker container that I want, as long as I map them properly OUTSIDE the container? Is there a particular book, website, or post you would recommend to read, to have a basic understanding concerning container paths, dockers, and perhaps even the linux file system? I don't have any illusions that I will be able to learn how to program in linux, but it sure would be nice for me to have a basic understanding of how directories are organized, the rules associated with how docker containers are configured, etc. so I can find things and be generally familiar with how to navigate around and 'follow the rules.' My knowledge right now feels pretty fragmented. I learn a lot from SpaceInvader One's videos, but there are a lot of gaps in my knowledge. For example, it caught me off guard at one point that using "/Music" as a path in docker could lead to a different outcome than specifying, for example, just "Music." Thanks again for your help!

Thanks, John_M and trurl. I realize that it was a band-aid solution, but I changed my docker size from 20gb to roughly 80gb. In doing so--and recognizing the sound advice in this thread--I knew that what I really needed to do was to identify what the problem was ... the trouble was that I didn't want the image to keep filling up while I was trying to identify the problem. In short, I was just trying to buy myself a little time. I have about 16 dockers installed, although usually only about 7 of them are running at any given time. trurl, I suspect the problem concerns how I configured my roonserver docker. Fix Common Problems identified two directories that differed only in case (Music vs. music). Additionally, I suspect that awhile back roon had been saving backups to the container rather than externally. Those backups are unfortunately all corrupt at this point, likely due to the above issue. I think I need to do some further investigation before reporting back, or I will just be wasting your time.

A quick note. I followed the instructions above, but the option to change the vdisk size never appeared. There must be some conditions that prevent the option from appearing (perhaps if your percentage of docker image utilization is too high???). I removed a couple of docker applications and rebooted my unraid machine, and then the option appeared and I was able to change the docker image size.

Ditto, thanks @Keek Uras! Dave

My NUC running Win 10 and Roon crashed so I thought I would give Roon a try on UnRAID, using your docker container (thanks!) I stumbled across this thread later, after having set things up. If I am reading this right, it sounds like if we use xthursdayx' updated template then we don't need to go through all the steps indicated above, so that things properly update when Roonlabs issues updates. How do I know if the updated template was present in Community Apps, when I installed? When I look at the roonserver entry in Community Applications, it says "Added to CA:September 19, 2020." Does this mean that the old template is still on CA? I'm not sure if the "added to" date indicated communicates when it originally appeared in CA, or if it communicates the latest version that was added. Also, a couple of quick public service announcements for UnRAID and docker newbies (I very much include myself in this category!) 1) MULTIPLE MUSIC DIRECTORIES: I have my music files in separate directories. The template by default provides a single entry for your music directory. I went ahead and entered one of my music directories, and then later I tried to add another directory using the main Roon interface (i.e., in the settings, storage area of Roon Remote). Needless to say, it didn't work after numerous attempts. After searching a bit I learned that docker containers can't really see your host file system, unless you map (mount?) particular directories. This is done through the aforementioned template, by selecting "Add another Path, Port, Variable, Label or Device," and then you just follow the formatting example established for the first music directory you set up. 2) ACCESSING THE ROON INTERFACE: This was noted elsewhere in this thread, but I thought I would enter it here as well. It is not necessary to add a port to access the Roon interface. Just install the container, and then with the Roon Remote software on your phone or laptop do a search for your Roon Core. It will find it, you can then access Roon settings etc. from your phone/laptop, and then you will be in music paradise. Thanks again for building this container! Dave

+1. I would love to see this.

I want to emphasize how exceptional these guides are. As a complete newbie to unraid and Linux, they are absolutely invaluable. I would never have been able to use unraid without them. I think I may have discovered the problem that results in receiving the following error: "Welcome to our server. The website is currently being setup under this address. For help and support, please contact: [email protected]." I encountered this when trying to setup nextcloud. I believe the issue is in the config php file. If you look at nextcloud.subdomain.conf.sample it provides setup instructions for the nextcloud config php file. These setup instructions include the following line, which didn't appear in the video: 'trusted_proxies' => ['letsencrypt'], In other words, config php should be as follows: 'trusted_proxies' => ['letsencrypt'], 'overwrite.cli.url' => 'https://nextcloud.your-domain.com/', 'overwritehost' => 'nextcloud.your-domain.com', 'overwriteprotocol' => 'https', The only other thing I did differently was to make absolutely sure that my subdomains were separated by commas (but no spaces!) in the letsencrypt docker container file. I tried some of the suggestions above, but unfortunately something got messed up along the way (I no longer even got the 'welcome' message). I had to delete the letsencrypt container and configure it over again to recover. One more thing I did (unrelated to the problem) was to make sure that all the relevant docker containers autostart (selecting the option on the main docker page). Hope this helps someone else. For my part, I am going to now go and make a donation to spaceinvaderone for how amazing this video was (and the preceding one on nextcloud). I now have a cloud server operating at home!!! Dave