GlennCottam

All is good, from what I can tell I was able to pull the data off the SSD. Took a while to finish wokring on a couple currupt databases, but other than that, everything went smoothly. Guess I need a new SSD. Thanks for the help!

Thats what I was afraid of. Thank you for the reply, I will see what data I am missing after restoring the data to the other 3 SSD's (hopefully none).

I have attempted to re-format the SSD's several times, but I am unable to format the NVME SSD I have installed. I tried formatting the cache in the configuration it was (all the SSD's in one cache), but decided I would try to seperate the NVME from the pool. I keep getting the error "Unmountable: Wrong or no file system" when I attempt to format the drive. In the syslog, I have the following entries when I attempt to format the drive: Apr 2 12:50:27 Unraid emhttpd: shcmd (37542): /sbin/wipefs -a /dev/nvme0n1 Apr 2 12:50:27 Unraid kernel: blk_update_request: critical medium error, dev nvme0n1, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0 Apr 2 12:50:28 Unraid kernel: blk_update_request: critical medium error, dev nvme0n1, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0 Apr 2 12:50:28 Unraid root: /dev/nvme0n1: 2 bytes were erased at offset 0x000001fe (dos): 55 aa Apr 2 12:50:28 Unraid root: /dev/nvme0n1: calling ioctl to re-read partition table: Success Apr 2 12:50:28 Unraid kernel: nvme0n1: p1 Apr 2 12:50:28 Unraid emhttpd: shcmd (37543): /sbin/wipefs -a /dev/nvme0n1p1 Apr 2 12:50:28 Unraid kernel: nvme0n1: p1 Apr 2 12:50:28 Unraid kernel: blk_update_request: critical medium error, dev nvme0n1, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0 Apr 2 12:50:28 Unraid kernel: blk_update_request: critical medium error, dev nvme0n1, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0 Apr 2 12:50:28 Unraid root: /dev/nvme0n1p1: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48 52 66 53 5f 4d Apr 2 12:50:28 Unraid emhttpd: shcmd (37544): mkfs.btrfs -f /dev/nvme0n1p1 Apr 2 12:50:28 Unraid kernel: blk_update_request: critical medium error, dev nvme0n1, sector 2048 op 0x3:(DISCARD) flags 0x800 phys_seg 1 prio class 0 Apr 2 12:50:28 Unraid kernel: blk_update_request: critical medium error, dev nvme0n1, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0 Apr 2 12:50:28 Unraid root: ERROR: error during mkfs: Operation not permitted Apr 2 12:50:28 Unraid root: btrfs-progs v5.15.1 Apr 2 12:50:28 Unraid root: See http://btrfs.wiki.kernel.org for more information. Apr 2 12:50:28 Unraid root: Apr 2 12:50:28 Unraid root: Performing full device TRIM /dev/nvme0n1p1 (447.13GiB) ... Apr 2 12:50:28 Unraid root: NOTE: several default settings have changed in version 5.15, please make sure Apr 2 12:50:28 Unraid root: this does not affect your deployments: Apr 2 12:50:28 Unraid root: - DUP for metadata (-m dup) Apr 2 12:50:28 Unraid root: - enabled no-holes (-O no-holes) Apr 2 12:50:28 Unraid root: - enabled free-space-tree (-R free-space-tree) Apr 2 12:50:28 Unraid root: Apr 2 12:50:28 Unraid emhttpd: shcmd (37544): exit status: 1 Apr 2 12:50:28 Unraid emhttpd: shcmd (37545): mkdir -p /mnt/nvme Apr 2 12:50:28 Unraid emhttpd: shcmd (37546): blkid -t TYPE='xfs' /dev/nvme0n1p1 &> /dev/null Apr 2 12:50:28 Unraid emhttpd: shcmd (37546): exit status: 2 Apr 2 12:50:28 Unraid emhttpd: shcmd (37547): blkid -t TYPE='btrfs' /dev/nvme0n1p1 &> /dev/null Apr 2 12:50:28 Unraid emhttpd: shcmd (37548): mount -t btrfs -o noatime,space_cache=v2 /dev/nvme0n1p1 /mnt/nvme Apr 2 12:50:28 Unraid kernel: BTRFS info (device nvme0n1p1): flagging fs with big metadata feature Apr 2 12:50:28 Unraid kernel: BTRFS info (device nvme0n1p1): using free space tree Apr 2 12:50:28 Unraid kernel: BTRFS info (device nvme0n1p1): has skinny extents Apr 2 12:50:28 Unraid root: mount: /mnt/nvme: wrong fs type, bad option, bad superblock on /dev/nvme0n1p1, missing codepage or helper program, or other error. Apr 2 12:50:28 Unraid emhttpd: shcmd (37548): exit status: 32 Apr 2 12:50:28 Unraid emhttpd: /mnt/nvme mount error: Wrong or no file sysem Apr 2 12:50:28 Unraid emhttpd: shcmd (37549): umount /mnt/nvme Apr 2 12:50:28 Unraid kernel: BTRFS error (device nvme0n1p1): devid 3 uuid 09a2f355-c5d8-46dc-94ad-3316672eda04 is missing Apr 2 12:50:28 Unraid kernel: BTRFS error (device nvme0n1p1): failed to read the system array: -2 Apr 2 12:50:28 Unraid kernel: BTRFS error (device nvme0n1p1): open_ctree failed I have attached a more recent diagnostics file. I can only assume that there is a physical issue with this SSD. I just hope when I pulled the data from the pool, the data on that drive was included. unraid-diagnostics-20220402-1257.zip

After re-reading the article, I apparently missed the command I was meant to use for 6.10+. After using the proper command, I was able to mount the SSD's. I take it from here, I just need to copy the data onto a drive, format the cache pool, and copy the data back? Thanks!

Came home today to find out the server can no longer mount the cache drives. All 4 of them that are in the pool. Booted into safemode, mounted the array, all other disks mount no problem, but the cache disks will not mount, giving the error: Unmountable: No File System. I came across this article: And attempted to mount the drive under /x, but then got the following error: mount: /x: can't read superblock on /dev/sdd1. This happens for all 4 drives I attempt to mount. There is a decent amount of data stored on these drives that I do need back. The drives should be fine, they have not been spitting out any errors the last few weeks, until this happened. Let me know if you require any more information! Any help is much appricated! unraid-diagnostics-20220331-2157.zip

Thank you very much for your responses! The issue has been resolved. I uninstalled the corefreq.plg plugin (I had forgotten I had it installed) and the server has had a uptime of 6 days.

Thank you so much! I have changed the BIOS "Power Supply Idle Control" to "Typical Current Idle" as my RAM was never set to overclock. Since the server seams to crash around 30 hours or so, I will follow up if it has successfully solved the issue.

I have been having various issues with Unraid for a little while now, and have been trying to hunt down what is causing the issue. I have tested every possible hardware-related issue the server could be having (new motherboard, swapped GPU's, even swapped CPU and RAM). I thought it may be my 10GbE network card that was causing issues, but I removed that with no change. I can't seem to figure out what is causing this. I have attempted to enable virtual machines with my Nvidia GPU a while ago, but I gave up and reverted everything the way it was (I think). Not sure if this could contribute to the issue. To no avail, I still am having issues. The server (usually a few hours after a parity-check is completed) will simply either lock up completely or kernel panic and reboot. The issue continues to happen when the server is in safe mode with all plugins disabled and minimal docker's running. Previously I have tried to uninstall problematic plugins (Nvidia driver, Tips and Tweaks etc...) but no change. Very rarely I actually get something in the syslog when the server crashes. The last kernel panic logged was on Feb 5 2022, but I had one today at 10:50am, with no errors saved in the log. The most latest logged crash is as follows: Feb 5 05:49:32 Unraid kernel: <IRQ> Feb 5 05:49:32 Unraid kernel: queued_spin_lock_slowpath+0x7/0xa Feb 5 05:49:32 Unraid kernel: nr_blockdev_pages+0x1d/0x6d Feb 5 05:49:32 Unraid kernel: si_meminfo+0x3f/0x5c Feb 5 05:49:32 Unraid kernel: Sys_MemInfo+0x25/0xa0 [corefreqk] Feb 5 05:49:32 Unraid kernel: ? tick_sched_do_timer+0x3e/0x3e Feb 5 05:49:32 Unraid kernel: ? update_cfs_rq_load_avg+0x117/0x125 Feb 5 05:49:32 Unraid kernel: ? Sys_DumpTask+0xed/0xf5 [corefreqk] Feb 5 05:49:32 Unraid kernel: Cycle_AMD_Family_17h+0x321/0x58b [corefreqk] Feb 5 05:49:32 Unraid kernel: ? SoC_RAPL+0x63/0x63 [corefreqk] Feb 5 05:49:32 Unraid kernel: ? SoC_RAPL+0x63/0x63 [corefreqk] Feb 5 05:49:32 Unraid kernel: Entry_AMD_F17h+0xb1/0xdd [corefreqk] Feb 5 05:49:32 Unraid kernel: ? Cycle_AMD_F17h+0x18/0x18 [corefreqk] Feb 5 05:49:32 Unraid kernel: __hrtimer_run_queues+0xfa/0x18a Feb 5 05:49:32 Unraid kernel: hrtimer_interrupt+0x92/0x160 Feb 5 05:49:32 Unraid kernel: __sysvec_apic_timer_interrupt+0x99/0xdb Feb 5 05:49:32 Unraid kernel: sysvec_apic_timer_interrupt+0x61/0x7d Feb 5 05:49:32 Unraid kernel: </IRQ> Feb 5 05:49:32 Unraid kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20 Feb 5 05:49:32 Unraid kernel: RIP: 0010:nr_blockdev_pages+0x48/0x6d Feb 5 05:49:32 Unraid kernel: Code: 8b 3d 7a 98 fa 00 48 8b 87 48 05 00 00 48 8d 97 48 05 00 00 48 2d 10 01 00 00 48 8d 88 10 01 00 00 48 39 d1 74 17 48 8b 48 30 <48> 8b 80 10 01 00 00 4c 03 61 58 48 2d 10 01 00 00 eb dd 48 81 c7 Feb 5 05:49:32 Unraid kernel: RSP: 0018:ffffc900032abe68 EFLAGS: 00000206 Feb 5 05:49:32 Unraid kernel: RAX: ffff888133a30398 RBX: ffffc900032abeb8 RCX: ffff888133a30508 Feb 5 05:49:32 Unraid kernel: RDX: ffff88810007a548 RSI: 000000000004b580 RDI: ffff88810007a000 Feb 5 05:49:32 Unraid kernel: RBP: ffffc900032abea8 R08: 00000000003d78dd R09: 0000000000000000 Feb 5 05:49:32 Unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000000008af Feb 5 05:49:32 Unraid kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 Feb 5 05:49:32 Unraid kernel: ? nr_blockdev_pages+0x1d/0x6d Feb 5 05:49:32 Unraid kernel: si_meminfo+0x3f/0x5c Feb 5 05:49:32 Unraid kernel: do_sysinfo.isra.0+0x9a/0x131 Feb 5 05:49:32 Unraid kernel: __do_sys_sysinfo+0x20/0x55 Feb 5 05:49:32 Unraid kernel: do_syscall_64+0x83/0xa5 Feb 5 05:49:32 Unraid kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae Feb 5 05:49:32 Unraid kernel: RIP: 0033:0x147546b45367 Feb 5 05:49:32 Unraid kernel: Code: f0 ff ff 73 01 c3 48 8b 0d fe 8a 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 b8 63 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d d1 8a 0c 00 f7 d8 64 89 01 48 Feb 5 05:49:32 Unraid kernel: RSP: 002b:00007ffc1724d148 EFLAGS: 00000206 ORIG_RAX: 0000000000000063 Feb 5 05:49:32 Unraid kernel: RAX: ffffffffffffffda RBX: 0000000000490f40 RCX: 0000147546b45367 Feb 5 05:49:32 Unraid kernel: RDX: 0000147546bd0024 RSI: 000000000000004c RDI: 00007ffc1724d150 Feb 5 05:49:32 Unraid kernel: RBP: 00007ffc1724d270 R08: 0000000000000000 R09: fffffffffffff800 Feb 5 05:49:32 Unraid kernel: R10: 0000147546a487f8 R11: 0000000000000206 R12: 0000000000000000 Feb 5 05:49:32 Unraid kernel: R13: 00000000004e350c R14: 0000000000000030 R15: 00000000000004f0 This panic happened a few times before the server locked up completely. I believe I have the kernel set to reboot automatically when it encounteres a error (since I am usually away from the server during the day), but it very rarely works. Please let me know if you need any more information! Any help is very much appreciated! unraid-diagnostics-20220207-1850.zip

That would make sense! Thank you!

Thank you! I formatted the drive, and then reformatted the cache pool. However, I now have a new problem. The pool is saying I only have 1.1TB of cache, when I have 2.1TB of SSD's installed. Everything that I can look at seams to be in working order. I am not sure where the issue lies. All the SSD's have the right allocation in their menus.

After moving the files all day yesterday, I am ready to try to repair the disk itself using: btrfs check --repair /dev/sdd I am wondering if using btrfs check --repair will erase the disk? On a side note when I checked this morning, the server was unresponsive. I couldn't access the physical terminal (keyboard and monitor on the server) as the screen was blank, the web ui would not respond, and my SSH connection failed. I had to force reboot the server to get control back. Unfortunately the syslog does not show anything useful. I have had these types of crashes before after I installed a new 10GbE networking card. Thought I had it fixed as it hasn't done this in a few weeks. I am not sure if this problem might be related to the cache disk, or another problem I have to find a solution for. Regardless, the single cache disk is still acting strange, still giving the same error, and in read only format.

Thanks for the quick reply! I did see that post in my googling but wanted to see if there was another option before I proceeded. Started the copy to the array. Ill reply if anything comes up.

Yesterday, I got a notification that the cache pool was full, and took action to rectify it (emptied my recycle bin, and invoked the mover, which cleared plenty of space off the pool). This morning, all of my dockers where down, and all dockers where saying the file system was read only. This meant my cache was mounted as read only. Thinking that one of the cache SSD's might have just been full, I attempted to balance the pool. The balance exited with: "BTRFS info (device sdd1): balance: ended with status -30". I rebooted the server, and attempted the balance again with the same error. The error only shows itself on the disk log for one of the SSD's. I have also tried a scrub which failed the same way. The other 3 SSD's have no errors in their logs, but the SSD having issues has the following errors on boot: Aug 19 13:00:33 Unraid kernel: BTRFS error (device sdd1): unable to find ref byte nr 11987553361920 parent 0 root 5 owner 36039698 offset 1277067264 Aug 19 13:00:33 Unraid kernel: BTRFS: error (device sdd1) in __btrfs_free_extent:3092: errno=-2 No such entry Aug 19 13:00:33 Unraid kernel: BTRFS info (device sdd1): forced readonly Aug 19 13:00:33 Unraid kernel: BTRFS: error (device sdd1) in btrfs_run_delayed_refs:2144: errno=-2 No such entry I then rebooted into safe mode, and entered maintenance mode in order to see if more information would show itself. I was able to run a BTRFS check on the SSD getting the following results: [1/7] checking root items [2/7] checking extents parent transid verify failed on 10084585357312 wanted 2949214 found 2949025 parent transid verify failed on 10957839237120 wanted 18446612689103786680 found 2993596 parent transid verify failed on 10957839237120 wanted 18446612689103786680 found 2993596 parent transid verify failed on 10957839237120 wanted 18446612689103786680 found 2993596 Ignoring transid failure data backref 11987553361920 root 5 owner 36039698 offset 1277067264 num_refs 0 not found in extent tree incorrect local backref count on 11987553361920 root 5 owner 36039698 offset 1277067264 found 1 wanted 0 back 0x11deca50 incorrect local backref count on 11987553361920 root 11763000595709952005 owner 4294936705 offset 1277067264 found 0 wanted 1 back 0x11df3a50 backref disk bytenr does not match extent record, bytenr=11987553361920, ref bytenr=0 backpointer mismatch on [11987553361920 86016] ERROR: errors found in extent allocation tree or chunk allocation [3/7] checking free space tree [4/7] checking fs roots parent transid verify failed on 10957839237120 wanted 18446612689103786680 found 2993596 Ignoring transid failure [5/7] checking only csums items (without verifying data) parent transid verify failed on 10957839237120 wanted 18446612689103786680 found 2993596 Ignoring transid failure [6/7] checking root refs [7/7] checking quota groups skipped (not enabled on this FS) ERROR: transid errors in file system Opening filesystem to check... Checking filesystem on /dev/sdd1 UUID: b81ae343-60cb-4303-8fac-192942135255 cache and super generation don't match, space cache will be invalidated found 1205836599296 bytes used, error(s) found total csum bytes: 1174204308 total tree bytes: 2446475264 total fs tree bytes: 917962752 total extent tree bytes: 130154496 btree space waste bytes: 430889587 file data blocks allocated: 1675174596608 referenced 1200826040320 I am completely lost on what to do at this point, and need assistance. The only thing I can think to do now is to copy the contents of the cache onto the array, and reformat the cache disks. I appreciate any help I can get! unraid-diagnostics-20210819-1322.zip