Arcaeus

Restarted system while working on another issue and the shares are showing up again? Why would they randomly disappear?

Attempted to repair the file system as I was getting local errors on the monitor attached to the bare metal computer. When trying to run 'xfs_repair -v /dev/md7' received this error: root@MediaVault:~# xfs_repair -v /dev/md7 Phase 1 - find and verify superblock... - block cache size set to 542376 entries Phase 2 - using internal log - zero log... zero_log: head block 159593 tail block 159588 ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this. Restarted the array regularly, and now disk 7 is showing 'Unmountable: not mounted' in Main - Array (attached).

Hey Jorge, I'm trying to figure out some share errors that aren't showing up now. When I ran 'ls -lah /mnt' it's showing question marks for disk 7 despite it showing ok in Main: root@MediaVault:~# ls -lah /mnt /bin/ls: cannot access '/mnt/disk7': Input/output error total 16K drwxr-xr-x 19 root root 380 May 6 10:54 ./ drwxr-xr-x 21 root root 480 May 10 09:58 ../ drwxrwxrwx 1 nobody users 80 May 6 11:04 cache/ drwxrwxrwx 9 nobody users 138 May 6 11:04 disk1/ drwxrwxrwx 8 nobody users 133 May 7 16:10 disk10/ drwxrwxrwx 5 nobody users 75 May 7 16:10 disk11/ drwxrwxrwx 6 nobody users 67 May 6 11:04 disk2/ drwxrwxrwx 9 nobody users 148 May 6 11:04 disk3/ drwxrwxrwx 4 nobody users 41 May 6 11:04 disk4/ drwxrwxrwx 10 nobody users 166 May 6 11:04 disk5/ drwxrwxrwx 7 nobody users 106 May 6 11:04 disk6/ d????????? ? ? ? ? ? disk7/ drwxrwxrwx 6 nobody users 73 May 7 16:10 disk8/ drwxrwxrwx 6 nobody users 67 May 6 11:04 disk9/ drwxrwxrwt 5 nobody users 100 May 6 13:41 disks/ drwxrwxrwt 2 nobody users 40 May 6 10:52 remotes/ drwxrwxrwt 2 nobody users 40 May 6 10:52 rootshare/ drwxrwxrwx 1 nobody users 138 May 6 11:04 user/ drwxrwxrwx 1 nobody users 138 May 6 11:04 user0/ I ran the file system check on disk 7 and got this output: entry ".." at block 0 offset 80 in directory inode 282079658 references non-existent inode 6460593193 entry ".." at block 0 offset 80 in directory inode 282079665 references non-existent inode 4315820253 entry "Season 1" in shortform directory 322311173 references non-existent inode 2234721490 would have junked entry "Season 1" in directory inode 322311173 entry "Season 2" in shortform directory 322311173 references non-existent inode 4579464692 would have junked entry "Season 2" in directory inode 322311173 entry "Season 3" in shortform directory 322311173 references non-existent inode 6453739819 would have junked entry "Season 3" in directory inode 322311173 entry "Season 5" in shortform directory 322311173 references non-existent inode 2234721518 would have junked entry "Season 5" in directory inode 322311173 entry "Season 6" in shortform directory 322311173 references non-existent inode 4760222352 would have junked entry "Season 6" in directory inode 322311173 would have corrected i8 count in directory 322311173 from 3 to 0 entry ".." at block 0 offset 80 in directory inode 322311206 references non-existent inode 6453779144 entry ".." at block 0 offset 80 in directory inode 322311221 references non-existent inode 6460505325 No modify flag set, skipping phase 5 Inode allocation btrees are too corrupted, skipping phases 6 and 7 No modify flag set, skipping filesystem flush and exiting. XFS_REPAIR Summary Tue May 10 10:06:15 2022 Phase Start End Duration Phase 1: 05/10 10:06:11 05/10 10:06:11 Phase 2: 05/10 10:06:11 05/10 10:06:11 Phase 3: 05/10 10:06:11 05/10 10:06:15 4 seconds Phase 4: 05/10 10:06:15 05/10 10:06:15 Phase 5: Skipped Phase 6: Skipped Phase 7: Skipped Total run time: 4 seconds Is there any reason to not run the file system check without the -n flag now? After that completes would I rebuild the drive like we did before or how does that work? New diags attached if needed. mediavault-diagnostics-20220510-1009.zip

I have Unraid exporting a few NFS shares to my Proxmox cluster, which are: Archives appdata ISOs Proxmox-VMs Proxmox-Backups Recently I've been having issues with the shares disconnecting on Proxmox. Research indicated that there was an issue with Mover, that when it runs it disconnects the NFS shares on Proxmox. So I made some of these shares either completely use the cache disk or never use it, 'Proxmox-VMs' being an example of the former and 'Proxmox-Backups' an example of the latter. I set up these shares, connected them to Proxmox, and they worked great. I checked them now after 24 hours and the Backups, ISOs, and VMs shares are disconnected in Proxmox. Archives and appdata are still connected no problem despite coming from the same location. When I check on the Unraid Dashboard, the shares are gone (see attached). However, when I open up the files on the cache disk, I see the 'Proxmox-VMs' folder right there. What would be causing these to not show up as shares? Diags attached. mediavault-diagnostics-20220509-2005.zip

Hey binhex, I just updated my HBA card firmware to the latest version and fixed some disk issues (noted in this thread: https://forums.unraid.net/topic/123025-chasing-down-disabled-drive-crc-errors/#comments) and now I'm receiving IO errors in SABnzbd: Failed making (/data/usenet_incomplete/*****) Traceback (most recent call last): File "/usr/lib/sabnzbd/sabnzbd/filesystem.py", line 719, in create_all_dirs os.mkdir(path_part_combined) OSError: [Errno 5] Input/output error: '/data/usenet_incomplete/*****' I changed nothing on my docker config or folders,and tried deleting and reinstalling the container (loading from template). When trying to change the file permissions in Krusader, it will just error out on a specific folder and wont let me move it elsewhere either. SABnzbd error log attached. Let me know what else you may need. SABnzbd IO error log.txt

Marking this issue as solved. Once the LSI 9207-8i was updated from FWVersion(20.00.00.00) to 20.00.07.00 (latest at this time), the CRC and IO errors stopped. The remaining posts were to resolve possible data corruption issues on my drives. Now SABnzbd is showing "OSError: [Errno 5] Input/output error: '/data/usenet_incomplete/...", and I opened a thread in the Binhex-SABnzbd channel here: https://forums.unraid.net/topic/44118-support-binhex-sabnzbd/?do=findComment&comment=1124052

So when I do that, it gives me this message saying that I would remove the preclear signature and it would have to re-clear the disk. I guess that doesn't matter as I'm not adding the disk into the array?

Here is what I'm seeing as I must be missing something:

Yep, looks like that got cleared out and so far staying at 1%. Alright, last question I have would be the process to format & mount the 2 blank 16TB drives currently sitting in UD. The idea here is that I would use these as a local backup of the data just in case something happens to the data on the array (like what almost happened here). While I wouldn't have parity in these, the plan would be to have a complete backup of everything that is kept in sync with Rclone or something similar, as well as a cloud backup offsite. Currently I can see the disks, but the Mount button is greyed out (I'm assuming due to them being precleared but not formatted yet). Destructive mode is enabled and UD Plus plugin is installed. Where do I go to format those? I'm assuming that I should just format them in XFS as I don't plan to remove them from my server, but does in make any sense to format them in NTFS? I saw this link that you had posted a few years back but it looks broken: https://forums.lime-technology.com/topic/44104-unassigned-devices-managing-disk-drives-outside-of-the-unraid-array/ . Is there an updated link or what are your thoughts on this idea and a process to complete it?

Alright, Parity sync has completed. Zero IO errors and zero CRC errors. On the dashboard, the log is showing 92% full. What log is that referring to and how do I check it or clean it out?

Understood. No CRC errors so far so seems like the firmware update did the trick. Will update this thread when the parity rebuild completes.

Alright, array has been started with that box unchecked. Parity-Sync/Data-Rebuild is in progress.

Disk is unmounted and assigned to the disk 5 slot. Array is not started yet. Earlier you mentioned backing up the contents of disk 5. Is that something we need to do before we start the array? Is the checkbox for rebuilding parity the "Parity is already valid" one? Which should be unchecked as we want to rebuild parity. Just wanted to confirm. What about disk 4? Will we handle that afterwords?

Sure, no problem. Would rather be safe than sorry. Here is what the main - array screen is showing right now. The old disk 5 is currently in UD and mounted.

Command ran again without the n, and nothing is in lost + found: root@MediaVault:~# xfs_repair -v /dev/sdk1 Phase 1 - find and verify superblock... - block cache size set to 542368 entries Phase 2 - using internal log - zero log... zero_log: head block 30 tail block 30 - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 1 - agno = 3 Phase 5 - rebuild AG headers and trees... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... XFS_REPAIR Summary Wed May 4 15:24:30 2022 Phase Start End Duration Phase 1: 05/04 15:24:20 05/04 15:24:20 Phase 2: 05/04 15:24:20 05/04 15:24:20 Phase 3: 05/04 15:24:20 05/04 15:24:27 7 seconds Phase 4: 05/04 15:24:27 05/04 15:24:27 Phase 5: 05/04 15:24:27 05/04 15:24:27 Phase 6: 05/04 15:24:27 05/04 15:24:30 3 seconds Phase 7: 05/04 15:24:30 05/04 15:24:30 Total run time: 10 seconds done What's next? Unmount and add back into array?

Yea, everything looks reasonably correct. At a glance nothing seems to be missing or broken. file system check completed. I don't see a lost & found folder on the drive so I'm assuming there weren't any errors? Although I'm assuming if this is just a check it's not moving anything. Here is the output from the command: root@MediaVault:~# xfs_repair -nv /dev/sdk1 Phase 1 - find and verify superblock... - block cache size set to 542368 entries Phase 2 - using internal log - zero log... zero_log: head block 20 tail block 20 - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 1 - agno = 0 - agno = 2 - agno = 3 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting. XFS_REPAIR Summary Wed May 4 15:02:47 2022 Phase Start End Duration Phase 1: 05/04 15:02:38 05/04 15:02:38 Phase 2: 05/04 15:02:38 05/04 15:02:38 Phase 3: 05/04 15:02:38 05/04 15:02:44 6 seconds Phase 4: 05/04 15:02:44 05/04 15:02:44 Phase 5: Skipped Phase 6: 05/04 15:02:44 05/04 15:02:47 3 seconds Phase 7: 05/04 15:02:47 05/04 15:02:47 Total run time: 9 seconds I still haven't seen any more errors, although the array has been stopped or in maintenance mode most of this whole time. Thus, I'm not sure if it's really had to do anything or has been communicating heavily with the drives. If it means anything, up until the disks started freaking out all of the files seemed reliable despite the large amount of CRC errors.

Yes? I thought this was the plan as JorgeB said the disk was probably fine. So then I should run "xfs_repair -v /dev/sdk1" right now to fix any file system errors on the disk? or run it with the -nv flag first?

So I have it mounted and am looking through the contents. Shouldn't I be looking at the emulated one to see what it will be applying to the disk? I agree, I just don't have another 4TB free. The only extra I have is the old disk 7. I have the 2x 16TBs in the server ready to go, they're just not formatted yet. These 2 disks are just sitting in UD and have been precleared. The plan was to just copy the files from the array directly to these disks as a local backup. Should I format these and copy the files from the original disk 5 to them? is that the "xfs_repair -v /dev/md5" command? How do I tell it that I want it on the UD drive? change md5 to sdk or something?

Yep, "Write corrections to parity disk" is set to yes. So I'm assuming that should be set to no then?

Ok preserve current assignments is done. I didn't see an option to do the file check on the drive without it being assigned in the array but it sounds like it should be fine without it. Now do I reassign the disk back into the array and let the rebuild begin? Then once it's complete look through the lost and found folder to see if there was anything that was missing? Or what is the right process here?

Not sure of the difference. It should just be the regular parity check that Unraid does that you can set the schedule to run for however long.

Sorry, still a bit confused here. I changed the UUID in UD, then mounted the old disk 5. Do I run the system filecheck again on that disk? Then do I need to do the Preserve current assignments part? And is that for both array and pool slots? I guess that's fine? does that data need to stay on there, or can/should it be deleted? Especially for disk 4 having 183GB, it seems like either it's useful and gets put back into the proper spot, or is not useful, the data is rebuilt on the disk, and then the lost + found data gets deleted to not take up unnecessary space.

It seems like the second option may be the better choice, as I'm not fully convinced that the actual disk 5 is corruption free and I don't have another new drive to copy to. Is there anything in particular that I need to consider before doing this? And does this mess anything up if disk 4 also has corruption on it?

New diags attached. I don't have any other 4TBs. I have the 2 16TBs in there that have been precleared, but that would be more than the 4TB parity so not sure how that would work. I still have the old disk 7 sitting on my desk that I haven't done anything with since I pulled it out for the rebuild. Does that help at all? As frowned upon as it may be, I did parity checks every week to make sure it was correct but not sure if that helps us right now. "comparing the data to the old one" meaning mount it and go folder by folder? And a new config being like a whole new Unraid setup or what? I have some of the files backed up but probably not all of them. My previous backup got corrupted and was in the process of creating a new one. mediavault-diagnostics-20220504-1319.zip

Array is started with disk 5 still being emulated. The lost and found folders are as follows: disk 4 - 5,824 objects: 66 directories, 5758 files (183 GB total) disk 5 - 811 objects: 602 directories, 209 files (2.49 GB total) That seems like a lot to me lol. What's next? Does this indicate that the disks are bad or that it could be caused by the CRC errors? Could I wipe/preclear the disks (one at a time) and then have them rebuilt, or what would you guys recommend?