Jump to content
kl0wn

Out Of Space Errors On Cache Drive

36 posts in this topic Last Reply

Recommended Posts

I do a balance every week to get around these issues as well in BTRFS.

Word of warning though... if one of your drives dies during a balance its goodbye data.

 

I am in the process now copying off my data, my second cache drive has an error where some the data is is kept toward the end of the drive.

If i power down the system and unplug Cache Drive 2 i can't see any data on Cache Drive 1.

When i plug both in i can see the filesystem and i have had the balance run through fully with no errors after deleting some files in what was newly created content.

It states that the data is raid1, but in actual fact when i power down the system again and unplug the drive with problems Cache 1 is missing some kind of index.

It appears BTRFS can only read Disk 1 if Disk 2 is there.

 

What a mess... so i am doing an rsync backup with both drives installed of my cache drive to a local disk.

Then ill format it to XFS on one Cache drive and copy it back. My god BTRFS is so Beta, it sounds great in theory but it doesn't seem to work that well in practice.

 

 

Share this post


Link to post
5 hours ago, wickedathletes said:

Anything specific stats people would need? Nothing will start for me.

 

See the instructions above, you may need to delete some files to get the balance started, besides that your docker image is corrupt, likely due to the cache being out space, you'll need to delete and recreate it:

 

Dec 19 20:40:00 Hades kernel: BTRFS warning (device loop2): loop2 checksum verify failed on 630734848 wanted DBC40929 found 5A858589 level 1
Dec 19 20:40:00 Hades kernel: BTRFS info (device loop2): read error corrected: ino 0 off 630734848 (dev /dev/loop2 sector 1248288)
Dec 19 20:40:00 Hades kernel: BTRFS info (device loop2): read error corrected: ino 0 off 630738944 (dev /dev/loop2 sector 1248296)
Dec 19 20:40:00 Hades kernel: BTRFS info (device loop2): read error corrected: ino 0 off 630743040 (dev /dev/loop2 sector 1248304)
Dec 19 20:40:00 Hades kernel: BTRFS info (device loop2): read error corrected: ino 0 off 630747136 (dev /dev/loop2 sector 1248312)

 

Share this post


Link to post
10 hours ago, kl0wn said:

EDIT: I'm a moron and didn't read your entire post. My balance took around 15 minutes. To be fair I never ran the operation and my SSD was in operation for 4 years so I'm unsure if that effects anything. However, I'm guessing yours may take a bit longer due to the size of the SSD.

 

My balance ran for awhile (I think an hour or so), but so far all has worked.

 

What was weird was obviously for the past 2 years it has run fine. Then it started in RC15, but it didn't happen until a few days into RC15, so it wasn't immediately apparent. Fingers crossed.

Share this post


Link to post
6 hours ago, johnnie.black said:

 

See the instructions above, you may need to delete some files to get the balance started, besides that your docker image is corrupt, likely due to the cache being out space, you'll need to delete and recreate it:

 


Dec 19 20:40:00 Hades kernel: BTRFS warning (device loop2): loop2 checksum verify failed on 630734848 wanted DBC40929 found 5A858589 level 1
Dec 19 20:40:00 Hades kernel: BTRFS info (device loop2): read error corrected: ino 0 off 630734848 (dev /dev/loop2 sector 1248288)
Dec 19 20:40:00 Hades kernel: BTRFS info (device loop2): read error corrected: ino 0 off 630738944 (dev /dev/loop2 sector 1248296)
Dec 19 20:40:00 Hades kernel: BTRFS info (device loop2): read error corrected: ino 0 off 630743040 (dev /dev/loop2 sector 1248304)
Dec 19 20:40:00 Hades kernel: BTRFS info (device loop2): read error corrected: ino 0 off 630747136 (dev /dev/loop2 sector 1248312)

 

 

Should I delete the image? It is working fine after the rebalance. I know recreating it is easy, but it is functioning normal.

Share this post


Link to post
8 minutes ago, wickedathletes said:

Should I delete the image? It is working fine after the rebalance. I know recreating it is easy, but it is functioning normal.

Those errors were correct, it means they were on metadata since the image uses DUP metadata profile, if all is working you can leave it, you can also scrub the docker image to make sure it's all good.

Share this post


Link to post
On 12/19/2017 at 12:36 PM, johnnie.black said:

this is because of the SSD allocation issue and it shouldn't happen anymore  on kernel 4.14

 

What was added in 4.14 kernel in this area?

Share this post


Link to post
 

What was added in 4.14 kernel in this area?

 

The allocation changed when the SSD mount mode is used (automatic when an SSD is detected) to try and avoid this issue:

 

v4.14 (Nov 2017)

  • added zstd compression
  • fine-grained check for degraded mount (verify raid constraints on chunk level, not device level)
  • userspace transaction ioctl has been deprecated, scheduled for removal in 4.17
  • foundation code for compression heuristics
  • mount option 'ssd' does not force block allocation alignments
  • fixes
    • potential raid repair and compression crash
    • prevent to set invalid default subvolid
    • resume qgroup rescan on rw remount
    • better reporting of detected checksum mismatches for DIO
    • compression for defrag vs per-file behaves as expected, respecting the requested value
    • possible deadlock with readdir and pagefault
    • emission of invalid clone operations in send
  • cleanups and refactoring

 

Share this post


Link to post
1 hour ago, johnnie.black said:

mount option 'ssd' does not force block allocation alignments

 

Saw that before, but didn't really know wtf that meant, thanks for the elucidation.

btw, next release includes YABFSCF (yet another btrfs free space calculation fix).

Share this post


Link to post
Posted (edited)

I just tried the command linked then it said done, moved one chunk. I tried to boot the VMs. They didnt boot and my VMs tab is gone in the GUI. I am having a panic attack because i dont know too much about this and I am afraid I just lost really important data on my vdisks. anybody please help

 

 

Edited by ColeAguirre

Share this post


Link to post

Go to Tools-diagnostics and attach the complete Diagnostics zip file to your next post. 

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.