BTRFS pool corrupt

[Jaster]

Hi Guys,

 

My BTRFS pool just went read-only.

 

Aug 10 22:28:49 Vortex kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 12, rd 11354, flush 1, corrupt 1, gen 0
Aug 10 22:28:49 Vortex kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 12, rd 11355, flush 1, corrupt 1, gen 0
Aug 10 22:28:50 Vortex kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 12, rd 11356, flush 1, corrupt 1, gen 0
Aug 10 22:28:50 Vortex kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 12, rd 11357, flush 1, corrupt 1, gen 0
Aug 10 22:28:50 Vortex kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 12, rd 11358, flush 1, corrupt 1, gen 0
Aug 10 22:28:50 Vortex kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 12, rd 11359, flush 1, corrupt 1, gen 0
Aug 10 22:28:51 Vortex kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 12, rd 11360, flush 1, corrupt 1, gen 0
Aug 10 22:28:51 Vortex kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 12, rd 11361, flush 1, corrupt 1, gen 0
Aug 10 22:28:51 Vortex kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 12, rd 11362, flush 1, corrupt 1, gen 0
Aug 10 22:28:51 Vortex kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 12, rd 11363, flush 1, corrupt 1, gen 0

 

Scrub does not seem to do anything.

 

a check -> btrfs dev stats -c /mnt/cache returns....

 

[/dev/nvme0n1p1].write_io_errs 0
[/dev/nvme0n1p1].read_io_errs 0
[/dev/nvme0n1p1].flush_io_errs 0
[/dev/nvme0n1p1].corruption_errs 0
[/dev/nvme0n1p1].generation_errs 0
[/dev/nvme1n1p1].write_io_errs 12
[/dev/nvme1n1p1].read_io_errs 12152
[/dev/nvme1n1p1].flush_io_errs 1
[/dev/nvme1n1p1].corruption_errs 1
[/dev/nvme1n1p1].generation_errs 0

 

SOS please

 

 

vortex-diagnostics-20210810-2230.zip

vortex-syslog-20210810-2033.zip

Edited August 10, 2021 by Jaster

[trurl]

 

[Jaster]

[Jaster]

WTF?! I tried to fix the btrfs issues and ran into this...

Please do NOT say "memtest" - the memory is fine, I tested it.

 

If I do boot in save mode, I don't get any panic... how can I track the issue down?!

Edited August 10, 2021 by Jaster

[trurl]

16 hours ago, Jaster said:

If I do boot in save mode, I don't get any panic... how can I track the issue down?!

SAFE mode doesn't load any plugins, so if it works in SAFE mode, then you need to figure out which plugin is to blame.

 

 

[JorgeB]

20 hours ago, Jaster said:

Please do NOT say "memtest" - the memory is fine, I tested it.

We ask to run memtest when checksum errors are detected, that's not your issue, your problem is that one of the NVMe devices dropped offline:

 

Aug 10 21:44:19 Vortex kernel: nvme nvme1: I/O 130 QID 1 timeout, aborting
Aug 10 21:44:19 Vortex kernel: nvme nvme1: Abort status: 0x0
Aug 10 21:44:26 Vortex kernel: nvme nvme1: I/O 183 QID 1 timeout, reset controller
Aug 10 21:45:33 Vortex kernel: nvme nvme1: I/O 15 QID 0 timeout, reset controller
Aug 10 21:46:27 Vortex kernel: nvme nvme1: Device not ready; aborting reset, CSTS=0x1

 

Reboot/power cycle to see if it comes back online and run a scrub, also see here for better pool monitoring.