Hi there, I had a power loss and restarted my server. Fix common problems popped up that the cache is read only. I did a parity check, still pops up. I've googled and see the common response is to backup and reformat the cache drives. I currently have 2 m.2s in a btrfs cache pool. Logs I'll attach but, i see a bunch of btrfs errors which im guessing means the fs is corrupt?

Jun 12 22:07:17 Tomservo kernel: BTRFS error (device nvme1n1p1): parent transid verify failed on 62775214080 wanted 162071 found 127225
Jun 12 22:07:17 Tomservo kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 0 off 62775214080 (dev /dev/nvme0n1p1 sector 850144)
Jun 12 22:07:17 Tomservo kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 0 off 62775218176 (dev /dev/nvme0n1p1 sector 850152)
Jun 12 22:07:17 Tomservo kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 0 off 62775222272 (dev /dev/nvme0n1p1 sector 850160)
Jun 12 22:07:17 Tomservo kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 0 off 62775226368 (dev /dev/nvme0n1p1 sector 850168)
Jun 12 22:07:17 Tomservo kernel: BTRFS error (device nvme1n1p1): parent transid verify failed on 62775377920 wanted 162071 found 127225
Jun 12 22:07:17 Tomservo kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 0 off 62775377920 (dev /dev/nvme0n1p1 sector 850464)
Jun 12 22:07:17 Tomservo kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 0 off 62775382016 (dev /dev/nvme0n1p1 sector 850472)
Jun 12 22:07:17 Tomservo kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 0 off 62775386112 (dev /dev/nvme0n1p1 sector 850480)
Jun 12 22:07:17 Tomservo kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 0 off 62775390208 (dev /dev/nvme0n1p1 sector 850488)
Jun 12 22:07:17 Tomservo kernel: BTRFS error (device nvme1n1p1): parent transid verify failed on 62776754176 wanted 162071 found 121946
Jun 12 22:07:17 Tomservo kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 0 off 62776754176 (dev /dev/nvme0n1p1 sector 853152)
Jun 12 22:07:17 Tomservo kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 0 off 62776758272 (dev /dev/nvme0n1p1 sector 853160)
Jun 12 22:07:17 Tomservo kernel: BTRFS error (device nvme1n1p1): parent transid verify failed on 62778687488 wanted 162071 found 127227
Jun 12 22:07:17 Tomservo kernel: BTRFS error (device nvme1n1p1): parent transid verify failed on 62778654720 wanted 162071 found 127225
Jun 12 22:07:17 Tomservo kernel: BTRFS error (device nvme1n1p1): parent transid verify failed on 62771642368 wanted 162071 found 127225
Jun 12 22:07:17 Tomservo kernel: BTRFS error (device nvme1n1p1): parent transid verify failed on 62772428800 wanted 162071 found 121946
Jun 12 22:07:17 Tomservo kernel: BTRFS error (device nvme1n1p1): parent transid verify failed on 62772412416 wanted 162071 found 121946
Jun 12 22:07:17 Tomservo kernel: BTRFS error (device nvme1n1p1): parent transid verify failed on 62772477952 wanted 162071 found 121943
Jun 12 22:07:17 Tomservo kernel: BTRFS error (device nvme1n1p1): parent transid verify failed on 62742544384 wanted 162066 found 127221

Could someone confirm and just sanity check me on what the proper backup and restore procedure would be, or if i have another option?
Thanks in advance!

 

syslog.txt

6 minutes ago, tomservo2814 said:

cache is read only. I did a parity check, still pops up.

Parity check has no effect on any data disks, and cache is outside the parity array anyway so no way parity check could help. 

 

Attach Diagnostics to your NEXT post in this thread 

15 minutes ago, trurl said:

Parity check has no effect on any data disks, and cache is outside the parity array anyway so no way parity check could help. 

 

Attach Diagnostics to your NEXT post in this thread 

you got it! Thanks!

tomservo-diagnostics-20230614-0915.zip

Jun 12 22:07:17 Tomservo kernel: BTRFS info (device nvme1n1p1): bdev /dev/nvme0n1p1 errs: wr 36013822, rd 3461405, flush 1294590, corrupt 0, gen 0

 

The above shows that on of the devices dropped offline in the past, reboot and see if you can run a correcting scrub, if it goes read-only again probably best to backup and re-format.

1 hour ago, JorgeB said:

Jun 12 22:07:17 Tomservo kernel: BTRFS info (device nvme1n1p1): bdev /dev/nvme0n1p1 errs: wr 36013822, rd 3461405, flush 1294590, corrupt 0, gen 0

 

The above shows that on of the devices dropped offline in the past, reboot and see if you can run a correcting scrub, if it goes read-only again probably best to backup and re-format.

apologies, how do i do a correcting scrub?

 

On the main GUI page click the first pool device device and scroll down to the scrub section.

2 minutes ago, JorgeB said:

On the main GUI page click the first pool device device and scroll down to the scrub section.

UUID:             6d502a25-fb1f-4eae-a144-0804736fe53a
Scrub started:    Wed Jun 14 11:09:33 2023
Status:           aborted
Duration:         0:00:00
Total to scrub:   153.80GiB
Rate:             0.00B/s
Error summary:    no errors found

sadly it just immediately aborts

In that case 

1 hour ago, JorgeB said:

probably best to backup and re-format.

 

1 minute ago, JorgeB said:

In that case 

 

im a sad panda, but thank you for guiding me through the journey. I never properly configured backup/restore appdata. Any guides for this process that you'd reccomend?

Since it's read-only, you should still be able to read (most of) its files, so you can just copy them to another disk.

 

You should disable Docker and VM Manager in Settings until you have cache working again.

On 6/14/2023 at 11:20 AM, trurl said:

Since it's read-only, you should still be able to read (most of) its files, so you can just copy them to another disk.

 

You should disable Docker and VM Manager in Settings until you have cache working again.

i shut down my machine and had to move apartments, finally set my machine back to try to do this, but rsync won't let me move anything because it says the source is read only . So am i fucked? I can see the files in the folder, so maybe its a permissions issue?

 

You can't move from a read-only source, since move means copy from source to destination, then delete from source. Delete is a write operation attempt on the read-only.

 

Try copying, not moving.

30 minutes ago, trurl said:

You can't move from a read-only source, since move means copy from source to destination, then delete from source. Delete is a write operation attempt on the read-only.

 

Try copying, not moving.

Sorry I realize that only said move. But I meant move and copy. Unfortunately neither does anything.

Post new diags after a copy attempt.

i just ended up starting from mostly scratch