Digaumspider Posted February 27, 2023 Share Posted February 27, 2023 Good day, all. I started seeing some erratic behaviour recently (after 2 years running just fine). Out of the blue, I started getting alerts to my email (I've configured it that way) from Fix Common Problems informing me that the SSD cache couldn't be written. Quote Event: Fix Common Problems - Unraid-Server Subject: Errors have been found with your server (Unraid-Server). Description: Investigate at Settings / User Utilities / Fix Common Problems Importance: alert * **Unable to write to cache-ssd** When clicking the Unraid Main page to confirm, though, here's what I see on my cache disks (they are 2 disks running as 1 cache in RAID1 mode): So there's plenty of space there. I also started noticing that PLEX wasn't working well, so I restarted the Docker image, but no dice. It keeps coming back up as unhealthy. The logs are full of BTRFS errors, but the SMART info for both SSDs running my cache are just dandy, so I have no idea what the heck is going on. I'm attaching the Logs here, in the hopes that someone may be able to point out what's the deal here, as I'm not really a pro at BTRFS by any stretch of the imagination and lately was just happy to keep my server running quietly there without any intervention (that's what I want my Unraid Server to do). Thank you all in advance. unraid-server-diagnostics-20230227-1822.zip Quote Link to comment
JorgeB Posted February 27, 2023 Share Posted February 27, 2023 Cache fs went read-only, reboot and post new diags after array start, together with the check filesystem output (default option, not in repair mode) Quote Link to comment
Digaumspider Posted February 27, 2023 Author Share Posted February 27, 2023 Attaching the logs here after a successful reboot. Also, the document you linked to has a piece of info that conflicts with my case (SS below). Here's what I see (it's telling me to start it in maintenance mode, although the instructions are clear that I should NEVER run this in maintenance mode for BTRFS filesystems): Any ideas? Thanks unraid-server-diagnostics-20230227-2138 (1).zip Quote Link to comment
JorgeB Posted February 28, 2023 Share Posted February 28, 2023 Scrub is run in normal mode, check filesystem which is want we want is in maintenance mode. Quote Link to comment
Digaumspider Posted February 28, 2023 Author Share Posted February 28, 2023 16 minutes ago, JorgeB said: Scrub is run in normal mode, check filesystem which is want we want is in maintenance mode. I'm sorry, did I misunderstand the paragraph below? I'm just trying to make sure I'm not going to destroy the data there by double checking. What am I missing? Quote If the file system is XFS or ReiserFS (but NOT BTRFS), then you must start the array in Maintenance mode, by clicking the Maintenance mode check box before clicking the Start button. This starts the unRAID driver but does not mount any of the drives. If the file system is BTRFS, then make sure the array is started, and NOT in Maintenance mode. Thanks again. Quote Link to comment
JorgeB Posted February 28, 2023 Share Posted February 28, 2023 Like mentioned and although it's not very clear that is talking about a scrub, not a filesystem check. Quote Link to comment
Digaumspider Posted February 28, 2023 Author Share Posted February 28, 2023 1 hour ago, JorgeB said: Like mentioned and although it's not very clear that is talking about a scrub, not a filesystem check. Thank you again for your patience, and I apologize for asking it again, but just trying to make sure that my data won't be wrecked, so please bear with me. I've clicked the link you posted, and it took me straight to that part of the post. I've got BTRFS running my cache, so in order to run said "Check Filesystem", what exactly should I do? I went over the instructions posted, but couldn't find anything that would indicate that I should bring the cache running BTRFS into maintenance mode and then run any different commands. Here's what I'm reading when I click the link you shared: Quote Checking and fixing drives in the webGui The instructions here are designed to check and fix the integrity of the file system of a data drive, while maintaining its parity info. Preparing to test The first step is to identify the file system of the drive you wish to test or repair. If you don't know for sure, then go to the Main page of the webGui, and click on the name of the drive (Disk 3, Cache, etc). Look for File system type, and you will see the file system format for your drive (should be xfs, reiserfs, or btrfs). If the file system is XFS or ReiserFS (but NOT BTRFS), then you must start the array in Maintenance mode, by clicking the Maintenance mode check box before clicking the Start button. This starts the unRAID driver but does not mount any of the drives. If the file system is BTRFS, then make sure the array is started, and NOT in Maintenance mode. Important note! Drives formatted with XFS or ReiserFS must be checked and repaired in Maintenance mode. Drives formatted with BTRFS cannot be checked or repaired in Maintenance mode! They must be checked and repaired with the array started in the regular mode. From the Main screen of the webGui, click the name of the disk that you want to test or repair. For example, if the drive of concern is Disk 5, then click on Disk 5. If it's the Cache drive, then click on Cache. You should see a page of options for that drive, beginning with various partition, file system format, and spin down settings. The section following that is the one you want, titled Check Filesystem Status. There is a box with the 2 words Not available in it. This is the command output box, where the progress and results of the command will be displayed. Below that is the Check button that starts the test or repair, followed by the options box where you can type in options for the test/repair command. The options box may already have default options in it, for a read-only check of the file system. For more help, click the Help button in the upper right. Running the test The default for most file system formats (XFS and ReiserFS) is a read-only check of the file system, with no changes made to the drive. If the file system is ReiserFS, then the options box will be blank initially, which implies the --check option. Always begin a ReiserFS drive test session with the --check option! If the file system is XFS, then the options box will contain the -n option (it means check only, no modification yet). We recommend adding the -v option (it means "verbose" for greater message display), so add a v to the -n, making it -nv. If the file system is BTRFS, then you want the scrub command, not the Balance command. The options box should already have the recommended options in it. Once the options are correct, click the Check button to start the operation. The progress and results will be displayed, and you will review them, and decide what action to take next. It is necessary to refresh the screen periodically, using the Refresh button underneath it or your browser's refresh button or keystroke, until the display indicates the test or repair is complete. If the results display a successful test, with no corruptions found, then you are done! Skip down to After the test and repair. Running the repair If however issues were found, the display of results will indicate the recommended action to take. Typically, that will involve repeating the command with a specific option, clearly stated, which you will type into the options box (including any hyphens, usually 2 leading hyphens). Then (if not BTRFS) click the Check button again to run the repair. Progress and results will again be displayed, and it's possible that you will have to run it again, with perhaps a different option. If the repairs were successful, it's recommended to run the read-only check one more time, to verify all is well now. For ReiserFS drives For more info on the reiserfsck tool and its options, see reiserfsck. There is also more information and some warnings about reiserfsck options in the Running reiserfsck section below. Important! If you are instructed to rerun the repair with the --rebuild-tree option, then PLEASE read the warnings in the Running reiserfsck section below! Important! If you are instructed to rerun the repair with the --rebuild-sb option, then you cannot continue here in the webGui! You will have to follow the instructions in the Drives formatted with ReiserFS using unRAID v5 or later section below. For XFS drives For more info on the xfs_repair tool and its options, see xfs_repair. See also the Drives formatted with XFS section below. For BTRFS drives and pools Use the scrub command. Use your best judgement, we have little experience yet. Do read the Drives formatted with BTRFS section below. For more info on the scrub command and its options, see btrfs scrub. Unfortunately, the scrub command can detect but not fix errors on single drives. If errors are detected, you will want to go to the Redoing a drive formatted with BTRFS section. If it's a drive pool, the scrub command can fix certain but not all errors. So far there is not a reliable BTRFS tool that can fix ALL possible errors. If the result indicates that there are still uncorrectable errors, then you will have to copy off all data and reformat the drive or pool anew (see Redoing a drive formatted with BTRFS). unRAID is committed to staying up-to-date with BTRFS development, so once better tools are ready, they will be available here too. During and at the conclusion of the command, a report will be displayed, in the command output box. If errors are detected, this report may specify additional actions to take. For XFS and BTRFS, we don't yet have much expertise to advise you. Use your best judgement. Most of us will probably do whatever it suggests we do. If your file system has only minor issues, then running the first option suggested should be all that is necessary. If it finds more significant corruption, then it may create a lost+found folder and place in it the files, folders, and parts of files it can recover. It will then be up to you to rename them and restore those files and folders to their correct locations. Many times, it will be possible to identify them by their contents and size. Note: If the repair command performs write operations to repair the file system, parity will be maintained. Thank you once again. Quote Link to comment
JorgeB Posted February 28, 2023 Share Posted February 28, 2023 Start the array in maintenance mode,click on cache, then click on "CHECK": Using the default settings, then post the output. Quote Link to comment
Digaumspider Posted February 28, 2023 Author Share Posted February 28, 2023 15 minutes ago, JorgeB said: Start the array in maintenance mode,click on cache, then click on "CHECK": Using the default settings, then post the output. Here you go: [1/7] checking root items [2/7] checking extents backref 1565130752 root 18446612693123346832 not referenced back 0x29cb4a0 tree backref 1565130752 root 2 not found in extent tree incorrect global backref count on 1565130752 found 2 wanted 1 backpointer mismatch on [1565130752 16384] ERROR: errors found in extent allocation tree or chunk allocation [3/7] checking free space tree [4/7] checking fs roots [5/7] checking only csums items (without verifying data) [6/7] checking root refs [7/7] checking quota groups skipped (not enabled on this FS) Opening filesystem to check... Checking filesystem on /dev/sdf1 UUID: e0fd7e5a-50c1-410a-89e6-f41a0c634d63 found 149333053440 bytes used, error(s) found total csum bytes: 145464004 total tree bytes: 377913344 total fs tree bytes: 171819008 total extent tree bytes: 34390016 btree space waste bytes: 57082664 file data blocks allocated: 3987076739072 referenced 145612578816 Quote Link to comment
Solution JorgeB Posted February 28, 2023 Solution Share Posted February 28, 2023 File system has issues, suggest backing up anything important then try to repair, or possibly better just format the pool and restore the data. Quote Link to comment
Digaumspider Posted March 1, 2023 Author Share Posted March 1, 2023 18 hours ago, JorgeB said: File system has issues, suggest backing up anything important then try to repair, or possibly better just format the pool and restore the data. I've followed the steps to format (and fully erase the content) of the pool of 2x SSD drives in btrfs (by changing to encrypted and then back to normal). Then, I created a new docker.img (recommended by the backup restoration tool), and started the docker service. I, then, restored the appdata from a back from last week, which took several hours. Even though I can see over 100GB of data (expected) in the appdata folder, including all my older docker applications, none of them show up under my docker area at all. So the data is physically there, but docker doesn't recognize them as it should... Since this is my first time restoring the data (I'm using the Backup/Restore appdata tool under settings), am I doing something wrong here? Diagnostics are attached, in case they're needed. I'll also be happy to post screenshots, if it helps. Thank you. unraid-server-diagnostics-20230301-1733.zip Quote Link to comment
itimpi Posted March 1, 2023 Share Posted March 1, 2023 41 minutes ago, Digaumspider said: I, then, restored the appdata from a back from last week, which took several hours This restores the docker containers variable data - it does not restore the docker containers themselves and the settings you used when installing them previously. How to do this (with their original settings intact) is covered here in the online documentation accessible via the Manual link at the bottom of the Unraid GUI or the Documentation link at the bottom of every forum page. Quote Link to comment
Digaumspider Posted March 1, 2023 Author Share Posted March 1, 2023 3 hours ago, itimpi said: This restores the docker containers variable data - it does not restore the docker containers themselves and the settings you used when installing them previously. How to do this (with their original settings intact) is covered here in the online documentation accessible via the Manual link at the bottom of the Unraid GUI or the Documentation link at the bottom of every forum page. Thanks. I wasn't aware of that. Now I know. I'll mark Jorge's response as the solution, though, as he allowed me to rebuild the cache and bring it back to a functional state again, to the point where I can use the docker solution normally again. Also, on the documentation: For what is worth, I tried following the steps there, but if you scroll up and take a look at my screenshot, the official documentation conflicts directly with the solution to the problem, so I thought it best to double (and triple) check here first before going ahead. Might be helpful to folks in the future if the documentation is updated to reflect the correct steps to be taken. Again, thank you for the help. Quote Link to comment
itimpi Posted March 1, 2023 Share Posted March 1, 2023 1 hour ago, Digaumspider said: Also, on the documentation: For what is worth, I tried following the steps there, but if you scroll up and take a look at my screenshot, the official documentation conflicts directly with the solution to the problem, so I thought it best to double (and triple) check here first before going ahead I am going to try and add more detail to the documentation to make it cleared how the different file system types should be handled from a check/repair perspective. I guess shortly we will have to add ZFS systems into this mix as well. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.