October 18, 20241 yr I'm spiraling out of control here haha... My server, seemingly from nowhere, started going down hill. Issues with Docker and libvert services not starting, getting corrupted, etc. I see a small number of SMART errors on a few of my disks, low numbers, 1 or 2. I have two m.2 SSD's selected to be used for Cache. I self diagnosed (in my head) that one of the SSD's that had some SMART errors may be going bad. I stopped the array, removed the disk (listed second in order) and started the array again. Now I'm getting an error - Unmountable: Unsupported or no file system Searching around, I found this command on another post, so I see there is still a partition there. I would really appreciate any assistance with getting it re-mounted. root@UNRAID:~# btrfs fi show Label: none uuid: 2aee4e2a-76da-4139-aa9d-a845993c2b52 Total devices 1 FS bytes used 380.00KiB devid 1 size 1.00GiB used 126.38MiB path /dev/loop2 warning, device 3 is missing Label: none uuid: 1f4efbb5-4661-41c7-8000-e098054e9447 Total devices 2 FS bytes used 206.96GiB devid 4 size 465.76GiB used 249.03GiB path /dev/nvme0n1p1 *** Some devices missing root@UNRAID:~# fdisk -l /dev/nvme0n1 Disk /dev/nvme0n1: 465.76 GiB, 500107862016 bytes, 976773168 sectors Disk model: Samsung SSD 970 EVO 500GB Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x00000000 Device Boot Start End Sectors Size Id Type /dev/nvme0n1p1 64 976773167 976773104 465.8G 83 Linux root@UNRAID:~# Diagnostics file attached. I'd certianly appreciate any assistance. If you see anything that points to why i'm having so many issues recently, that would be great. I'm just tyring to get the data off this thing (appdata specifically) so I can migrate. unraid-diagnostics-20241018-1243.zip
October 19, 20241 yr Community Expert The problem appears to be the Nvme device: Oct 18 12:19:17 UNRAID kernel: critical medium error, dev nvme0n1, sector 216008768 op 0x1:(WRITE) flags 0x1800 phys_seg 20 prio class 2 Do you still have the other device?
October 19, 20241 yr Author Hi JorgeB, thank you for the reply. Yes, it is still installed in the machine after I removed it from the pool. It currently sits in Unassigned Disk Devices. Prior to my initial post, I did look at pulling out the one you point out with the medium error, and add this other device back, but it states "All existing data on this device will be OVERWRITTEN when array is Started" That's where I stopped. Is there a way to get the array started with just using this second device and keep the data? Just poking around, I was wondering if I could set the slot to 1 and put in just the second devic and start it ok. The gui at least that way does not say it will be erased.
October 19, 20241 yr Community Expert Stop array, unassign all poo devices, start array, stop array, reassign only sdd, start array, post new diags.
October 19, 20241 yr Author Ok, done. The disk added, the gui afterwards says it is unmountable. Diags uploaded. Thank you! unraid-diagnostics-20241019-1317.zip
October 19, 20241 yr Community Expert There's no valid filesystem on that device, it was probably wiped before, type: sfdisk /dev/sdd then type 64 and hit Enter, don't do anything else and post the output you see.
October 20, 20241 yr Author It was not wiped, it was a member of the Cache disk pool until removed by just stopping the array, setting the line for that disk to be blank, and starting it again. Here is the output from that command; root@UNRAID:~# sfdisk /dev/sdd Welcome to sfdisk (util-linux 2.38.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Checking that no-one is using this disk right now ... OK Disk /dev/sdd: 465.76 GiB, 500107862016 bytes, 976773168 sectors Disk model: Samsung SSD 850 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes sfdisk is going to create a new 'dos' disk label. Use 'label: <name>' before you define a first partition to override the default. Type 'help' to get more information. >>>
October 21, 20241 yr Community Expert 8 hours ago, jbritt1234 said: It was not wiped, it was a member of the Cache disk pool until removed by just stopping the array, setting the line for that disk to be blank, and starting it again. That will wipe it. You didn't create the new parition: On 10/20/2024 at 12:53 AM, JorgeB said: then type 64 and hit Enter, don't do anything else and post the output you see.
October 21, 20241 yr Author 5 hours ago, JorgeB said: That will wipe it. You didn't create the new parition: Well, crap, ok. I didn't realize that. Sorry about that next step, i got a bit wrapped up in the "don't do anythign else" part out of fear hahah. root@UNRAID:~# sfdisk /dev/sdd Welcome to sfdisk (util-linux 2.38.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Checking that no-one is using this disk right now ... OK Disk /dev/sdd: 465.76 GiB, 500107862016 bytes, 976773168 sectors Disk model: Samsung SSD 850 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes sfdisk is going to create a new 'dos' disk label. Use 'label: <name>' before you define a first partition to override the default. Type 'help' to get more information. >>> 64 Created a new DOS disklabel with disk identifier 0x6dce7c90. Created a new partition 1 of type 'Linux' and of size 465.8 GiB. Partition #1 contains a btrfs signature. Do you want to remove the signature? [Y]es/[N]o:
October 21, 20241 yr Community Expert Type No and Enter to keep the signature Then write and Enter to write the changes Then output of btrfs fi show
October 21, 20241 yr Author Ok, here is the output. root@UNRAID:~# sfdisk /dev/sdd Welcome to sfdisk (util-linux 2.38.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Checking that no-one is using this disk right now ... OK Disk /dev/sdd: 465.76 GiB, 500107862016 bytes, 976773168 sectors Disk model: Samsung SSD 850 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes sfdisk is going to create a new 'dos' disk label. Use 'label: <name>' before you define a first partition to override the default. Type 'help' to get more information. >>> 64 Created a new DOS disklabel with disk identifier 0xd99f4ba1. Created a new partition 1 of type 'Linux' and of size 465.8 GiB. Partition #1 contains a btrfs signature. Do you want to remove the signature? [Y]es/[N]o: no /dev/sdd1 : 64 976773167 (465.8G) Linux /dev/sdd2: write New situation: Disklabel type: dos Disk identifier: 0xd99f4ba1 Device Boot Start End Sectors Size Id Type /dev/sdd1 64 976773167 976773104 465.8G 83 Linux The partition table has been altered. Calling ioctl() to re-read partition table. Syncing disks. root@UNRAID:~# btrfs fi show Label: none uuid: 2aee4e2a-76da-4139-aa9d-a845993c2b52 Total devices 1 FS bytes used 380.00KiB devid 1 size 1.00GiB used 126.38MiB path /dev/loop2 Label: none uuid: 1f4efbb5-4661-41c7-8000-e098054e9447 Total devices 2 FS bytes used 206.96GiB devid 3 size 465.76GiB used 249.03GiB path /dev/sdd1 devid 4 size 465.76GiB used 249.03GiB path /dev/nvme0n1p1 root@UNRAID:~#
October 21, 20241 yr Community Expert Solution Now unassign both pool devices, start array, stop array, re-assign both pools devices, start array and post new diags.
October 21, 20241 yr Author Ok, done! I do see the pool started and there is data! I'm attaching the diagnostics. I do wonder the state of the nvme0n1 device. If i need to pull it out to avoid corruption (with docker, etc) again, etc. TY sooo much for your help so far. unraid-diagnostics-20241021-1111.zip
October 21, 20241 yr Community Expert Pool is raid1, so now try to remove the nvme device which is still showing errors, stop the array, unassign the nvme, start the array.
October 21, 20241 yr Author Ok. That's exactly what I did prior. I stopped it again. I Changed the nvme to No device. I checked the "Yes, I want to do this" box, and started it again. I'm now back to the sdd device showing as unmountable: Unsupported or no file system. So, should we go back through that again, but this time not add the nvme back at all, but keep the cache as a single disk with just the sdd? When this kicked off, I was getting lots of issues with the docker and VM service not running correctly, and seemingly corrupting some files.
October 22, 20241 yr Author For the time being, I went back through the same steps and have it back with both disks mounted. I'm going to try to get some of the appdata info FTP'd off the server as the CABackup app stopped working so my backup is nearly 2 years old. I'm a bit worried that that disk will cause issues with corruption, etc. I guess if you can think of a good way to just mount the sdd disk only, that would be good. I can just re-copy again to ensure it is good data. Thanks for your help so far.
October 22, 20241 yr Community Expert It's a good idea to try and make some backups, but docker and VM images may already be corrupt, I had noticed that in the previous diags, but try to backup what you can.
October 22, 20241 yr Author I ended up pulling the physical nvme disk to simulate a failure. After that, the array started ok and I've been successfull getting most of the data. Thank you Jorge. 🍻
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.