I do a balance every week to get around these issues as well in BTRFS.

Word of warning though... if one of your drives dies during a balance its goodbye data.

 

I am in the process now copying off my data, my second cache drive has an error where some the data is is kept toward the end of the drive.

If i power down the system and unplug Cache Drive 2 i can't see any data on Cache Drive 1.

When i plug both in i can see the filesystem and i have had the balance run through fully with no errors after deleting some files in what was newly created content.

It states that the data is raid1, but in actual fact when i power down the system again and unplug the drive with problems Cache 1 is missing some kind of index.

It appears BTRFS can only read Disk 1 if Disk 2 is there.

 

What a mess... so i am doing an rsync backup with both drives installed of my cache drive to a local disk.

Then ill format it to XFS on one Cache drive and copy it back. My god BTRFS is so Beta, it sounds great in theory but it doesn't seem to work that well in practice.

 

 

5 hours ago, wickedathletes said:

Anything specific stats people would need? Nothing will start for me.

 

See the instructions above, you may need to delete some files to get the balance started, besides that your docker image is corrupt, likely due to the cache being out space, you'll need to delete and recreate it:

 

Dec 19 20:40:00 Hades kernel: BTRFS warning (device loop2): loop2 checksum verify failed on 630734848 wanted DBC40929 found 5A858589 level 1
Dec 19 20:40:00 Hades kernel: BTRFS info (device loop2): read error corrected: ino 0 off 630734848 (dev /dev/loop2 sector 1248288)
Dec 19 20:40:00 Hades kernel: BTRFS info (device loop2): read error corrected: ino 0 off 630738944 (dev /dev/loop2 sector 1248296)
Dec 19 20:40:00 Hades kernel: BTRFS info (device loop2): read error corrected: ino 0 off 630743040 (dev /dev/loop2 sector 1248304)
Dec 19 20:40:00 Hades kernel: BTRFS info (device loop2): read error corrected: ino 0 off 630747136 (dev /dev/loop2 sector 1248312)

 

10 hours ago, kl0wn said:

EDIT: I'm a moron and didn't read your entire post. My balance took around 15 minutes. To be fair I never ran the operation and my SSD was in operation for 4 years so I'm unsure if that effects anything. However, I'm guessing yours may take a bit longer due to the size of the SSD.

 

My balance ran for awhile (I think an hour or so), but so far all has worked.

 

What was weird was obviously for the past 2 years it has run fine. Then it started in RC15, but it didn't happen until a few days into RC15, so it wasn't immediately apparent. Fingers crossed.

6 hours ago, johnnie.black said:

 

See the instructions above, you may need to delete some files to get the balance started, besides that your docker image is corrupt, likely due to the cache being out space, you'll need to delete and recreate it:

 

Dec 19 20:40:00 Hades kernel: BTRFS warning (device loop2): loop2 checksum verify failed on 630734848 wanted DBC40929 found 5A858589 level 1
Dec 19 20:40:00 Hades kernel: BTRFS info (device loop2): read error corrected: ino 0 off 630734848 (dev /dev/loop2 sector 1248288)
Dec 19 20:40:00 Hades kernel: BTRFS info (device loop2): read error corrected: ino 0 off 630738944 (dev /dev/loop2 sector 1248296)
Dec 19 20:40:00 Hades kernel: BTRFS info (device loop2): read error corrected: ino 0 off 630743040 (dev /dev/loop2 sector 1248304)
Dec 19 20:40:00 Hades kernel: BTRFS info (device loop2): read error corrected: ino 0 off 630747136 (dev /dev/loop2 sector 1248312)

 

 

Should I delete the image? It is working fine after the rebalance. I know recreating it is easy, but it is functioning normal.

8 minutes ago, wickedathletes said:

Should I delete the image? It is working fine after the rebalance. I know recreating it is easy, but it is functioning normal.

Those errors were correct, it means they were on metadata since the image uses DUP metadata profile, if all is working you can leave it, you can also scrub the docker image to make sure it's all good.

On 12/19/2017 at 12:36 PM, johnnie.black said:

this is because of the SSD allocation issue and it shouldn't happen anymore  on kernel 4.14

 

What was added in 4.14 kernel in this area?

2 minutes ago, limetech said:

 

What was added in 4.14 kernel in this area?

https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-Linux-4.14

 

What was added in 4.14 kernel in this area?

 

The allocation changed when the SSD mount mode is used (automatic when an SSD is detected) to try and avoid this issue:

 

v4.14 (Nov 2017)

• added zstd compression
• fine-grained check for degraded mount (verify raid constraints on chunk level, not device level)
• userspace transaction ioctl has been deprecated, scheduled for removal in 4.17
• foundation code for compression heuristics
• mount option 'ssd' does not force block allocation alignments
• fixes
  • potential raid repair and compression crash
  • prevent to set invalid default subvolid
  • resume qgroup rescan on rw remount
  • better reporting of detected checksum mismatches for DIO
  • compression for defrag vs per-file behaves as expected, respecting the requested value
  • possible deadlock with readdir and pagefault
  • emission of invalid clone operations in send
• cleanups and refactoring

 

1 hour ago, johnnie.black said:

mount option 'ssd' does not force block allocation alignments

 

Saw that before, but didn't really know wtf that meant, thanks for the elucidation.

btw, next release includes YABFSCF (yet another btrfs free space calculation fix).

I just tried the command linked then it said done, moved one chunk. I tried to boot the VMs. They didnt boot and my VMs tab is gone in the GUI. I am having a panic attack because i dont know too much about this and I am afraid I just lost really important data on my vdisks. anybody please help

 

 

Edited October 6, 20196 yr by ColeAguirre

Go to Tools-diagnostics and attach the complete Diagnostics zip file to your next post. 

Had this exact issue today, I've recently changed from a Raid0 w/2 ssd-drives to a single ssd. Had to reformat my cache drive to fix this as mentioned earlier in this thread. Strange issue but whatever, it works again !  

 

Ramblings; 

Just had to delete my docker.img file and redownload the dockers (you'll get a notice about this when you try to enable your docker service after the reformat and backup restore)

 

Oh and just a side note, I took a backup of the data with: rsync -a --progress /mnt/cache/ /mnt/disk1/temp  before formatting of course. (reverse when copying back).

And just a note, plex database files failed to copy, i suspect since the cache got "full" it somehow corrupted the db-files. Thankfully my plex takes its own db backups so just had to roll back to latest backup by renaming the affected .db files, all good now.