February 28, 20242 yr I tried to replace the cache devices by bigger and faster ones. For this I tried to follow to procedure described in the Unraid 6 FAQ. Now I am unable to access the data on the cache drives. What I did trying to replace the drive 1. Add the two new SSDs to the system 2. Stop the array 3. Replace the device in the pool 4. Start the array 5. I then saw the error "Unmountable: Unsupported or no file system" Trying to fix it, I did the following: 1. Stop the array 2. Remove the newly added device from the pool 3. I still got the error "Unmountable: Unsupported or no file system" Now I am trying to restore the data from the cache drive What I see in the UI currently: Overview over devices ("ls -la /dev/sd*"): brw-rw---- 1 root plugdev 8, 0 Feb 27 12:05 /dev/sda brw-rw---- 1 root plugdev 8, 1 Feb 27 12:05 /dev/sda1 brw-rw---- 1 root disk 8, 16 Feb 27 22:36 /dev/sdb brw-rw---- 1 root disk 8, 17 Feb 27 22:46 /dev/sdb1 brw-rw---- 1 root disk 8, 32 Feb 27 22:35 /dev/sdc brw-rw---- 1 root disk 8, 48 Feb 27 12:05 /dev/sdd brw-rw---- 1 root disk 8, 49 Feb 27 12:05 /dev/sdd1 brw-rw---- 1 root disk 8, 64 Feb 27 12:05 /dev/sde brw-rw---- 1 root disk 8, 65 Feb 27 12:05 /dev/sde1 brw-rw---- 1 root disk 8, 80 Feb 27 12:05 /dev/sdf brw-rw---- 1 root disk 8, 81 Feb 27 12:05 /dev/sdf1 Note: There is no /dev/sdc1 Trying to run "btrfs restore -d /dev/sdb1 /mnt/user/btrfs_restore/" shows: warning, device 2 is missing bad tree block 386624946176, bytenr mismatch, want=386624946176, have=0 ERROR: cannot read chunk root Could not open root, trying backup super warning, device 2 is missing bad tree block 386624946176, bytenr mismatch, want=386624946176, have=0 ERROR: cannot read chunk root Could not open root, trying backup super warning, device 2 is missing bad tree block 386624946176, bytenr mismatch, want=386624946176, have=0 ERROR: cannot read chunk root Could not open root, trying backup super Trying to run "btrfs rescue zero-log /dev/sdb1" shows: warning, device 2 is missing ERROR: cannot read chunk root ERROR: could not open ctree Trying to run "btrfs-select-super -s 1 /dev/sdb1" shows: warning, device 2 is missing ERROR: cannot read chunk root ERROR: could not open ctree Trying to run "btrfs-select-super -s 1 /dev/sdb1" shows: Opening filesystem to check... warning, device 2 is missing bad tree block 386624946176, bytenr mismatch, want=386624946176, have=0 ERROR: cannot read chunk root ERROR: cannot open file system Trying to restore partition /dev/sdc1 Hoping to be able to make the missing device accessible again, I tried to use "gdisk" on /dev/sdc, but the output is confusing me: "gdisk /dev/sdc" shows: GPT fdisk (gdisk) version 1.0.9.1 Partition table scan: MBR: not present BSD: not present APM: not present GPT: not present Creating new GPT entries in memory. Command (? for help): q "gdisk /dev/sdb" shows: GPT fdisk (gdisk) version 1.0.9.1 Partition table scan: MBR: MBR only BSD: not present APM: not present GPT: not present *************************************************************** Found invalid GPT and valid MBR; converting MBR to GPT format in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by typing 'q' if you don't want to convert your MBR partitions to GPT format! *************************************************************** Warning! Secondary partition table overlaps the last partition by 33 blocks! You will need to delete this partition or resize it in another utility. Command (? for help): q Questions What did I do wrong on trying to replace the cache devices? Do you have an idea, how I could be able to restore the data from the cache drives?
February 28, 20242 yr Community Expert 2 hours ago, Adrian Dymorz said: What did I do wrong on trying to replace the cache devices? If you didn't yet reboot please post the diagnostics
February 28, 20242 yr Author 44 minutes ago, JorgeB said: If you didn't yet reboot please post the diagnostics I did reboot because that is what sometimes helps too.
February 28, 20242 yr Community Expert 54 minutes ago, Adrian Dymorz said: I did reboot In that case cannot see what happened, but post the output of btrfs fi show sdb and sdc were the original pool members, correct?
February 28, 20242 yr Author "btrfs fi show" returns Label: none uuid: ed9ea52f-f8c5-40d9-9814-16f41b39251d Total devices 2 FS bytes used 35.00GiB devid 1 size 1.82TiB used 37.01GiB path /dev/nvme0n1p1 devid 2 size 1.82TiB used 37.01GiB path /dev/nvme1n1p1 Label: none uuid: fb0b5703-1252-43c3-8ee5-d49187894ad9 Total devices 1 FS bytes used 14.90MiB devid 1 size 35.00GiB used 1.52GiB path /dev/loop2 Label: none uuid: c8d7555b-394a-4039-9ea1-ddeaa04a6785 Total devices 1 FS bytes used 440.00KiB devid 1 size 1.00GiB used 126.38MiB path /dev/loop3 warning, device 2 is missing ERROR: cannot read chunk root Label: none uuid: b7d22b34-fe12-4ec1-aa33-cadfd0a480e4 Total devices 2 FS bytes used 380.41GiB devid 1 size 465.76GiB used 424.00GiB path /dev/sdb1 *** Some devices missing Yes, sdb and sdc were the original pool members.
February 28, 20242 yr Community Expert type sfdisk /dev/sdc then type 2048 and hit enter, post the output of that
February 28, 20242 yr Author "sfdisk /dev/sdc" returns Welcome to sfdisk (util-linux 2.38.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Checking that no-one is using this disk right now ... OK Disk /dev/sdc: 465.76 GiB, 500107862016 bytes, 976773168 sectors Disk model: WDC WDS500G2B0A Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes sfdisk is going to create a new 'dos' disk label. Use 'label: <name>' before you define a first partition to override the default. Type 'help' to get more information. >>> "2048" as input returned: Created a new DOS disklabel with disk identifier 0xf50eb252. Created a new partition 1 of type 'Linux' and of size 465.8 GiB. /dev/sdc1 : 2048 976773167 (465.8G) Linux Now it is waiting for more input: /dev/sdc2:
February 28, 20242 yr Community Expert Hit CTRL + C to abort, then repeat the same but now type 64 instead of 2048.
February 28, 20242 yr Author >>> 64 Created a new DOS disklabel with disk identifier 0x9a67f18e. Created a new partition 1 of type 'Linux' and of size 465.8 GiB. Partition #1 contains a btrfs signature. Do you want to remove the signature? [Y]es/[N]o: Edited February 28, 20242 yr by Adrian Dymorz
February 28, 20242 yr Community Expert Solution Type N and enter to keep the signature, then type write and enter, post new output of btrfs fi show
February 28, 20242 yr Author Output of "sfdisk /dev/sdc" and the inputs "64", "N" and "write": Welcome to sfdisk (util-linux 2.38.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Checking that no-one is using this disk right now ... OK Disk /dev/sdc: 465.76 GiB, 500107862016 bytes, 976773168 sectors Disk model: WDC WDS500G2B0A Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes sfdisk is going to create a new 'dos' disk label. Use 'label: <name>' before you define a first partition to override the default. Type 'help' to get more information. >>> 64 Created a new DOS disklabel with disk identifier 0xf84ac41d. Created a new partition 1 of type 'Linux' and of size 465.8 GiB. Partition #1 contains a btrfs signature. Do you want to remove the signature? [Y]es/[N]o: N /dev/sdc1 : 64 976773167 (465.8G) Linux /dev/sdc2: write New situation: Disklabel type: dos Disk identifier: 0xf84ac41d Device Boot Start End Sectors Size Id Type /dev/sdc1 64 976773167 976773104 465.8G 83 Linux The partition table has been altered. Calling ioctl() to re-read partition table. Syncing disks. Output of "btrfs fi show": Label: none uuid: ed9ea52f-f8c5-40d9-9814-16f41b39251d Total devices 2 FS bytes used 35.00GiB devid 1 size 1.82TiB used 37.01GiB path /dev/nvme0n1p1 devid 2 size 1.82TiB used 37.01GiB path /dev/nvme1n1p1 Label: none uuid: fb0b5703-1252-43c3-8ee5-d49187894ad9 Total devices 1 FS bytes used 14.90MiB devid 1 size 35.00GiB used 1.52GiB path /dev/loop2 Label: none uuid: c8d7555b-394a-4039-9ea1-ddeaa04a6785 Total devices 1 FS bytes used 440.00KiB devid 1 size 1.00GiB used 126.38MiB path /dev/loop3 Label: none uuid: b7d22b34-fe12-4ec1-aa33-cadfd0a480e4 Total devices 2 FS bytes used 380.41GiB devid 1 size 465.76GiB used 424.00GiB path /dev/sdb1 devid 2 size 465.76GiB used 423.03GiB path /dev/sdc1
February 28, 20242 yr Community Expert You should now be able to add the two old devices to a new pool and start the array to mount it, if it doesn't munt post diags after array start.
February 28, 20242 yr Author I currently run "btrfs restore -d /dev/sdb1 /mnt/user/btrfs_restore/" to have a copy of the data before I do anything else. After that I am going to try to add the devices to a new pool. Thanks for your help! Is the link posted on the Unraid Community Rockstars page still valid? Edited February 28, 20242 yr by Adrian Dymorz
February 28, 20242 yr Author Creating a copy of the data and adding the devices to a new pool did work out flawlessly. Not sure about the root cause of the issue. That will probably remain an unsolved mystery.
February 28, 20242 yr Community Expert 8 minutes ago, Adrian Dymorz said: Not sure about the root cause of the issue. Based on the results and the description, looks like Unraid ran wipefs on that device but then it failed to replace it with the new device, but why it failed, I could only say if the diags from when it first happened were available.
March 31, 20242 yr Hello, I'm in a very similar situation as Adrian here where after trying to replace a cache drive one of the 2 drives in the pool seams to have been wiped completely and the other is unmountable with the following being generated in the log: Mar 30 23:10:09 Vader-1 unassigned.devices: Mounting partition 'sdk1' at mountpoint '/mnt/disks/TPBF2209290060500231'... Mar 30 23:10:09 Vader-1 unassigned.devices: Mount cmd: /sbin/mount -t 'btrfs' -o ro,relatime,space_cache=v2,discard '/dev/sdk1' '/mnt/disks/TPBF2209290060500231' Mar 30 23:10:10 Vader-1 unassigned.devices: Mount of 'sdk1' failed: 'mount: /mnt/disks/TPBF2209290060500231: wrong fs type, bad option, bad superblock on /dev/sdk1, missing codepage or helper program, or other error. dmesg(1) may have more information after failed mount system call. ' Mar 30 23:10:10 Vader-1 unassigned.devices: Partition 'TPBF2209290060500231' cannot be mounted. sdk and sdl were the original pool members. "btrfs fi show" returns: warning, device 2 is missing warning, device 2 is missing ERROR: cannot read chunk root Label: none uuid: 488a4ce6-34fe-420f-a8ef-9e5862e79a28 Total devices 2 FS bytes used 45.95GiB devid 1 size 953.87GiB used 45.00GiB path /dev/sdk1 *** Some devices missing Label: none uuid: 7f4fbf85-9803-4a4f-a951-3a6b45414213 Total devices 1 FS bytes used 144.00KiB devid 1 size 953.87GiB used 2.02GiB path /dev/sdl1 "sfdisk /dev/sdk" returns: Welcome to sfdisk (util-linux 2.38.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Checking that no-one is using this disk right now ... OK Disk /dev/sdk: 953.87 GiB, 1024209543168 bytes, 2000409264 sectors Disk model: TEAM T2531TB Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0xb78f201f Old situation: Device Boot Start End Sectors Size Id Type /dev/sdk1 2048 2000409263 2000407216 953.9G 83 Linux Type 'help' to get more information. >>> Thank you in advance and any help at all would be greatly appreciated.
March 31, 20242 yr Community Expert 4 hours ago, PerryS said: "sfdisk /dev/sdk" returns: sdk is good, what was the other pool device? That's the one missing.
March 31, 20242 yr 38 minutes ago, JorgeB said: sdk is good, what was the other pool device? That's the one missing. Sdl was the second pool device. It mounts fine but contains no data.
March 31, 20242 yr Community Expert 10 minutes ago, PerryS said: Sdl was the second pool device. That means the device was just not wiped, but also reformatted, I'm sorry but I can't help recovering from that, only if the device was just wiped.
March 31, 20242 yr 3 hours ago, JorgeB said: That means the device was just not wiped, but also reformatted, I'm sorry but I can't help recovering from that, only if the device was just wiped. It makes sense to me that I wouldn’t be able to restore the data on sdl but is there not a way to just get sdk to mount? As I understand it being that it was a two device pool wouldn’t that mean the entirety of the data was stored on both disks? So if there is a way to mount sdk then I could backup the contents and the rebuild the pool all together.
April 1, 20242 yr Community Expert 17 hours ago, PerryS said: As I understand it being that it was a two device pool wouldn’t that mean the entirety of the data was stored on both disks? If the pool was redundant you can mount it with the remaining device, to do that just create a new pool with 2 slots and assign only sdk, start the array, if the pool doesn't mount post the diagnostics.
April 1, 20242 yr 12 hours ago, JorgeB said: If the pool was redundant you can mount it with the remaining device, to do that just create a new pool with 2 slots and assign only sdk, start the array, if the pool doesn't mount post the diagnostics. Adding the one device (sdk) to the cache pool and starting the array yields this: I am attaching the diagnostics export below. vader-1-diagnostics-20240401-1703.zip
April 2, 20242 yr Community Expert 11 hours ago, PerryS said: Adding the one device (sdk) to the cache pool and starting the array yields this: Apr 1 15:03:21 Vader-1 kernel: BTRFS error (device sdk1): failed to read chunk root This suggests the pool wasn't redundant, at least no fully, so it won't mount without the missing device.
April 3, 20242 yr 14 hours ago, JorgeB said: Apr 1 15:03:21 Vader-1 kernel: BTRFS error (device sdk1): failed to read chunk root This suggests the pool wasn't redundant, at least no fully, so it won't mount without the missing device. That is supper bizarre as it has always been set up as a redundant pool with 2 drives. Any idea what might have caused this to happen? I'm assuming whatever happened happened during the process of tying to replace one of the cache drives. But I followed the portion of the Official Unraid Manual to the letter when I did. Is there any chance of recovering the data off of that drive at this point or am I just completely out of luck? Also thank you so much for your time @JorgeB. I really appreciate all of the advice.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.