October 20, 20241 yr I hope someone can help, please? I woke up this morning to find my cache pool of 2 devices were no longer mounted. I've tried stopping the array, un-assign both cache devices. Start array. Stop array, reassign both cache devices. Start array. = Same error. Please advice! box-diagnostics-20241020-1546.zip Edited October 21, 20241 yr by Xenu
October 20, 20241 yr Community Expert Pool is complaining of a missing device, post the output from btrfs fi show
October 21, 20241 yr Author Thank you for the reply. This is the response: root@box:~# btrfs fi show Label: none uuid: 94418e24-71a8-4933-a795-612a5f71ebf5 Total devices 1 FS bytes used 412.00KiB devid 1 size 2.00GiB used 228.75MiB path /dev/loop2 warning, device 2 is missing warning, device 2 is missing ERROR: cannot read chunk root Label: none uuid: adb84329-04d6-471b-afbd-2443872bcda4 Total devices 2 FS bytes used 1.88TiB devid 1 size 1.82TiB used 1.82TiB path /dev/sdi1 *** Some devices missing This is with the array online and both cache drives assigned.
October 21, 20241 yr Community Expert Any idea why there's a missing device? Did you try replacing or removing cache2 before?
October 21, 20241 yr Author No idea. Nothing was tinkered with/changed, just happened overnight. Checked cables, all fine there.
October 21, 20241 yr Community Expert Stop the array and type and post the output from fdisk -l /dev/sdi and fdisk -l /dev/sdd
October 21, 20241 yr Author 3 minutes ago, JorgeB said: Stop the array and type and post the output from fdisk -l /dev/sdi and fdisk -l /dev/sdd root@box:~# fdisk -l /dev/sdi Disk /dev/sdi: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors Disk model: CT2000MX500SSD1 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x00000000 Device Boot Start End Sectors Size Id Type /dev/sdi1 2048 3907029167 3907027120 1.8T 83 Linux root@box:~# fdisk -l /dev/sdd Disk /dev/sdd: 476.94 GiB, 512110190592 bytes, 1000215216 sectors Disk model: Tech Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes I notice the two drives are now also showing under historical devices:
October 21, 20241 yr Community Expert cache2 is missing the partition, type sfdisk /dev/sdd then type 2048 and ENTER and post the output from that, without typing anything else
October 21, 20241 yr Author 1 minute ago, JorgeB said: cache2 is missing the partition, type sfdisk /dev/sdd then type 2048 and ENTER and post the output from that, without typing anything else Thank you. root@box:~# sfdisk /dev/sdd Welcome to sfdisk (util-linux 2.38.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Checking that no-one is using this disk right now ... OK Disk /dev/sdd: 476.94 GiB, 512110190592 bytes, 1000215216 sectors Disk model: Tech Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes sfdisk is going to create a new 'dos' disk label. Use 'label: <name>' before you define a first partition to override the default. Type 'help' to get more information. >>> 2048 Created a new DOS disklabel with disk identifier 0x7ad5d360. Created a new partition 1 of type 'Linux' and of size 476.9 GiB. /dev/sdd1 : 2048 1000215215 (476.9G) Linux /dev/sdd2: Looks like it's awaiting another input?
October 21, 20241 yr Community Expert Hit CTRL + C to abort, then repeat but use 64 instead of 2048, and post output of that.
October 21, 20241 yr Author 28 minutes ago, JorgeB said: Hit CTRL + C to abort, then repeat but use 64 instead of 2048, and post output of that. root@box:~# sfdisk /dev/sdd Welcome to sfdisk (util-linux 2.38.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Checking that no-one is using this disk right now ... OK Disk /dev/sdd: 476.94 GiB, 512110190592 bytes, 1000215216 sectors Disk model: Tech Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes sfdisk is going to create a new 'dos' disk label. Use 'label: <name>' before you define a first partition to override the default. Type 'help' to get more information. >>> 64 Created a new DOS disklabel with disk identifier 0xcc0341ce. Created a new partition 1 of type 'Linux' and of size 476.9 GiB. Partition #1 contains a btrfs signature. Do you want to remove the signature? [Y]es/[N]o:
October 21, 20241 yr Community Expert Solution Type No and Enter to keep the signature Then type write and Enter to save the changes Output of btrfs fi show
October 21, 20241 yr Author 15 minutes ago, JorgeB said: Type No and Enter to keep the signature Then type write and Enter to save the changes Output of btrfs fi show Do you want to remove the signature? [Y]es/[N]o: N /dev/sdd1 : 64 1000215215 (476.9G) Linux /dev/sdd2: write New situation: Disklabel type: dos Disk identifier: 0xcc0341ce Device Boot Start End Sectors Size Id Type /dev/sdd1 64 1000215215 1000215152 476.9G 83 Linux The partition table has been altered. Calling ioctl() to re-read partition table. Syncing disks. root@box:~# btrfs fi show Label: none uuid: adb84329-04d6-471b-afbd-2443872bcda4 Total devices 2 FS bytes used 1.88TiB devid 1 size 1.82TiB used 1.82TiB path /dev/sdi1 devid 2 size 476.94GiB used 476.94GiB path /dev/sdd1
October 21, 20241 yr Author 13 minutes ago, JorgeB said: Now start the array and post new diags. You genius, functioning again! Thank you! I manually stopped Docker in case there's any ongoing issues. box-diagnostics-20241021-1609.zip Edited October 21, 20241 yr by Xenu
October 21, 20241 yr Community Expert There's still the question of what happened to the partition, but maybe it was a fluke, take the opportunity to update your backups if needed.
October 21, 20241 yr Author Just now, JorgeB said: There's still the question of what happened to the partition, but maybe it was a fluke, take the opportunity to update your backups if needed. Many many thanks, Jorge! Popped you a little beer money your way!
October 21, 20241 yr Community Expert Thanks! There's also data corruption being detected by btrfs, you should run a scrub.
October 21, 20241 yr Author 4 minutes ago, JorgeB said: Thanks! There's also data corruption being detected by btrfs, you should run a scrub. Running it now, already started making corrections. Thank you again!
October 22, 20241 yr Author 20 hours ago, JorgeB said: Thanks! There's also data corruption being detected by btrfs, you should run a scrub. Hello Jorge, thank you again! I left the scrub running overnight and immediately it was finding bad bits. It did however abort the scrub by itself overnight and attempting the scrub again simply does nothing (pressing the srub button had the page pending for a second or two, but nothing more happened). I powed off the server. Booting it back up today, the cache pool has also reverted to how it looked at the start of this thread. I had a go at bringing the cache pool back online using your steps above, as the terminal response from each step was the same, however after reaching this step: - I'm presented with this: root@box:~# btrfs fi show Label: none uuid: 94418e24-71a8-4933-a795-612a5f71ebf5 Total devices 1 FS bytes used 412.00KiB devid 1 size 2.00GiB used 228.75MiB path /dev/loop2 warning, device 2 is missing warning, device 2 is missing ERROR: cannot read chunk root Label: none uuid: adb84329-04d6-471b-afbd-2443872bcda4 Total devices 2 FS bytes used 1.88TiB devid 1 size 1.82TiB used 1.82TiB path /dev/sdi1 *** Some devices missing Am I right to assume one of the drives needs replacing, as it's damaged/on it way out? Is there any way to identify what drive of the two might be the cause? And if so, is there anyway to shift the cache from one drive onto the other, or both cache drives content onto the non-cache drives until I can replace the cache drives? Let me know if you feel I need to make a new thread for this! box-diagnostics-20241022-1236.zip Edited October 22, 20241 yr by Xenu
October 22, 20241 yr Community Expert Looks like cache2 lost the partition again, this is not normal, was the device always connect with USB? Some USB bridges can cause strange issues. In the meantime post the output from fdisk -l /dev/sdd
October 22, 20241 yr Author 1 hour ago, JorgeB said: Looks like cache2 lost the partition again, this is not normal, was the device always connect with USB? Some USB bridges can cause strange issues. In the meantime post the output from fdisk -l /dev/sdd Thank you again for your time! root@box:~# fdisk -l /dev/sdd Disk /dev/sdd: 476.94 GiB, 512110190592 bytes, 1000215216 sectors Disk model: Tech Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Yes, one of the drives has always been connected via USB. I've been meaning to migrate back to one cache drive, internal SSD. If it's the USB one that's causing problems/damaged, would you be able to offer advice on how to migrate the data over and remove the USB drive from the pool, please? If I can get the array back online, I'll gladly just moved everything off the cache for time being. Edited October 22, 20241 yr by Xenu
October 22, 20241 yr Community Expert Type sfdisk /dev/sdd then type 64 and Enter and post the results
October 22, 20241 yr Author 22 minutes ago, JorgeB said: Type sfdisk /dev/sdd then type 64 and Enter and post the results root@box:~# sfdisk /dev/sdd Welcome to sfdisk (util-linux 2.38.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Checking that no-one is using this disk right now ... OK Disk /dev/sdd: 476.94 GiB, 512110190592 bytes, 1000215216 sectors Disk model: Tech Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes sfdisk is going to create a new 'dos' disk label. Use 'label: <name>' before you define a first partition to override the default. Type 'help' to get more information. >>>
October 22, 20241 yr Community Expert 47 minutes ago, JorgeB said: then type 64 and Enter and post the results
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.