Hi,

 

Came home to a cache disk showing as unmountable yesterday and have been trying to troubleshoot the issue. Would love any help finding out if it's recoverable or a goner. 

minilith-diagnostics-20221216-1801.zip

Edited December 16, 20223 yr by nerval
Added diagnostics file.

Attach Diagnostics to your NEXT post in this thread 

Dec 16 17:53:58 Minilith kernel: critical medium error, dev nvme0n1, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0
Dec 16 17:53:58 Minilith kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 1, corrupt 0, gen 0
Dec 16 17:53:58 Minilith kernel: BTRFS warning (device nvme0n1p1): chunk 1048576 missing 1 devices, max tolerance is 0 for writable mount

 

Device problem, and be because it cannot write to it btrfs cannot mount read/write, you can try mounting read-only with the UD plugin to see if the problem is with writes only, if yes that would allow for data recovery.

40 minutes ago, trurl said:

Attach Diagnostics to your NEXT post in this thread 

 

Planned on doing that but got a message I'd hit my limit of posts for the day so thought adding it to the OP was the best solution to that. Will remember to post it in a separate post in the future though.

 

31 minutes ago, JorgeB said:

Dec 16 17:53:58 Minilith kernel: critical medium error, dev nvme0n1, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0
Dec 16 17:53:58 Minilith kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 1, corrupt 0, gen 0
Dec 16 17:53:58 Minilith kernel: BTRFS warning (device nvme0n1p1): chunk 1048576 missing 1 devices, max tolerance is 0 for writable mount

 

Device problem, and be because it cannot write to it btrfs cannot mount read/write, you can try mounting read-only with the UD plugin to see if the problem is with writes only, if yes that would allow for data recovery.

 

Seems that's a no-go for data recovery. Appreciate the help though! 

 

Dec 16 18:40:56 Minilith unassigned.devices: Adding partition 'nvme0n1p1'...
Dec 16 18:40:56 Minilith unassigned.devices: Mounting partition 'nvme0n1p1' at mountpoint '/mnt/disks/50026B7282104CC2'...
Dec 16 18:40:56 Minilith unassigned.devices: Mount drive command: /sbin/mount -t 'btrfs' -o rw,noatime,nodiratime,space_cache=v2,discard '/dev/nvme0n1p1' '/mnt/disks/50026B7282104CC2'
Dec 16 18:40:56 Minilith kernel: BTRFS info (device nvme0n1p1): turning on sync discard
Dec 16 18:40:56 Minilith kernel: BTRFS info (device nvme0n1p1): using free space tree
Dec 16 18:40:56 Minilith kernel: BTRFS info (device nvme0n1p1): has skinny extents
Dec 16 18:40:56 Minilith kernel: BTRFS info (device nvme0n1p1): enabling ssd optimizations
Dec 16 18:40:56 Minilith kernel: BTRFS info (device nvme0n1p1): start tree-log replay
Dec 16 18:40:56 Minilith kernel: nvme0n1: I/O Cmd(0x0) @ LBA 18446744073709551615, 0 blocks, I/O Error (sct 0x1 / sc 0x82) 
Dec 16 18:40:56 Minilith kernel: critical medium error, dev nvme0n1, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0
Dec 16 18:40:56 Minilith kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 1, corrupt 0, gen 0
Dec 16 18:40:56 Minilith kernel: BTRFS warning (device nvme0n1p1): chunk 1048576 missing 1 devices, max tolerance is 0 for writable mount
Dec 16 18:40:56 Minilith kernel: BTRFS: error (device nvme0n1p1) in write_all_supers:4368: errno=-5 IO failure (errors while submitting device barriers.)
Dec 16 18:40:56 Minilith kernel: BTRFS warning (device nvme0n1p1: state E): Skipping commit of aborted transaction.
Dec 16 18:40:56 Minilith kernel: BTRFS: error (device nvme0n1p1: state EA) in cleanup_transaction:1982: errno=-5 IO failure
Dec 16 18:40:56 Minilith kernel: BTRFS: error (device nvme0n1p1: state EA) in btrfs_replay_log:2500: errno=-5 IO failure (Failed to recover log tree)
Dec 16 18:40:56 Minilith kernel: BTRFS error (device nvme0n1p1: state EA): open_ctree failed
Dec 16 18:40:56 Minilith unassigned.devices: Mount of 'nvme0n1p1' failed: 'mount: /mnt/disks/50026B7282104CC2: can't read superblock on /dev/nvme0n1p1.        dmesg(1) may have more information after failed mount system call. '
Dec 16 18:40:56 Minilith unassigned.devices: Partition '50026B7282104CC2' cannot be mounted.

 

1 minute ago, nerval said:

Seems that's a no-go

It's not mounting read only, you need to enable that in UD for that disk.

18 minutes ago, JorgeB said:

It's not mounting read only, you need to enable that in UD for that disk.

 

Thanks for the heads up, tested with read only enabled now and didn't seem to work either. That's a bummer.

 

Dec 16 19:04:31 Minilith unassigned.devices: Adding partition 'nvme0n1p1'...
Dec 16 19:04:31 Minilith unassigned.devices: Mounting partition 'nvme0n1p1' at mountpoint '/mnt/disks/50026B7282104CC2'...
Dec 16 19:04:31 Minilith unassigned.devices: Mount drive command: /sbin/mount -t 'btrfs' -o ro,noatime,nodiratime,space_cache=v2,discard '/dev/nvme0n1p1' '/mnt/disks/50026B7282104CC2'
Dec 16 19:04:31 Minilith kernel: BTRFS info (device nvme0n1p1): turning on sync discard
Dec 16 19:04:31 Minilith kernel: BTRFS info (device nvme0n1p1): using free space tree
Dec 16 19:04:31 Minilith kernel: BTRFS info (device nvme0n1p1): has skinny extents
Dec 16 19:04:31 Minilith kernel: BTRFS info (device nvme0n1p1): enabling ssd optimizations
Dec 16 19:04:31 Minilith kernel: BTRFS info (device nvme0n1p1): start tree-log replay
Dec 16 19:04:31 Minilith kernel: nvme0n1: I/O Cmd(0x0) @ LBA 18446744073709551615, 0 blocks, I/O Error (sct 0x1 / sc 0x82) 
Dec 16 19:04:31 Minilith kernel: critical medium error, dev nvme0n1, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0
Dec 16 19:04:31 Minilith kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 1, corrupt 0, gen 0
Dec 16 19:04:31 Minilith kernel: BTRFS warning (device nvme0n1p1): chunk 1048576 missing 1 devices, max tolerance is 0 for writable mount
Dec 16 19:04:31 Minilith kernel: BTRFS: error (device nvme0n1p1) in write_all_supers:4368: errno=-5 IO failure (errors while submitting device barriers.)
Dec 16 19:04:31 Minilith kernel: BTRFS warning (device nvme0n1p1: state E): Skipping commit of aborted transaction.
Dec 16 19:04:31 Minilith kernel: BTRFS: error (device nvme0n1p1: state EA) in cleanup_transaction:1982: errno=-5 IO failure
Dec 16 19:04:31 Minilith kernel: BTRFS: error (device nvme0n1p1: state EA) in btrfs_replay_log:2500: errno=-5 IO failure (Failed to recover log tree)
Dec 16 19:04:31 Minilith kernel: BTRFS error (device nvme0n1p1: state EA): open_ctree failed
Dec 16 19:04:31 Minilith unassigned.devices: Mount of 'nvme0n1p1' failed: 'mount: /mnt/disks/50026B7282104CC2: can't read superblock on /dev/nvme0n1p1.        dmesg(1) may have more information after failed mount system call. '
Dec 16 19:04:31 Minilith unassigned.devices: Partition '50026B7282104CC2' cannot be mounted.

 

You can try manually here, using the rescue=all,ro option, but not sure it will be any different.

2 hours ago, JorgeB said:

You can try manually here, using the rescue=all,ro option, but not sure it will be any different.

 

Thanks, got the following error message when trying that:

 

mount: /temp: wrong fs type, bad option, bad superblock on /dev/nvme0n1, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.

 

Log says the following:

 

Dec 16 21:55:52 Minilith kernel: squashfs: Unknown parameter 'rescue'
Dec 16 21:55:52 Minilith kernel: fuseblk: Unknown parameter 'rescue'
Dec 16 21:55:52 Minilith kernel: UDF-fs: bad mount option "rescue=all" or missing value
Dec 16 21:55:52 Minilith kernel: xfs: Unknown parameter 'rescue'

 

Edited December 16, 20223 yr by nerval
Added info from log.

11 hours ago, nerval said:

/dev/nvme0n1

It's missing the partition,  should be /dev/nvme0n1p1

32 minutes ago, JorgeB said:

It's missing the partition,  should be /dev/nvme0n1p1

 

Thank you. Well, that at least didn't give an error message in the log. Having a hard time sorting out what to do from there though. Used the command "mkdir /recovertemp" but can't seem to locate anything by that name anywhere.

 

Dec 17 09:32:32 Minilith kernel: BTRFS info (device nvme0n1p1): enabling all of the rescue options
Dec 17 09:32:32 Minilith kernel: BTRFS info (device nvme0n1p1): ignoring data csums
Dec 17 09:32:32 Minilith kernel: BTRFS info (device nvme0n1p1): ignoring bad roots
Dec 17 09:32:32 Minilith kernel: BTRFS info (device nvme0n1p1): disabling log replay at mount time
Dec 17 09:32:32 Minilith kernel: BTRFS info (device nvme0n1p1): using free space tree
Dec 17 09:32:32 Minilith kernel: BTRFS info (device nvme0n1p1): has skinny extents
Dec 17 09:32:32 Minilith kernel: BTRFS info (device nvme0n1p1: state C): enabling ssd optimizations

 

Post the output of

btrfs fi usage -T /recovertemp

 

Yeah, doesn't seem to have created the folder from what I can gather. 

 

root@Minilith:~# btrfs fi usage -T /recoverytemp
ERROR: cannot access '/recoverytemp': No such file or directory

 

Hmm, ls shows the folders alongside the other folders but with a green square. I can browse through the folders and it seems to be the correct cache drive folders from what I can gather. 

Edited December 17, 20223 yr by nerval
Additional info.

19 minutes ago, nerval said:

/recoverytemp

before you said it was /recovertemp

Well, I'll be damned. Did not notice I made a type there. Thanks! Now the result gives a lot more sense. 

 

Overall:
    Device size:                 447.13GiB
    Device allocated:            335.02GiB
    Device unallocated:          112.11GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                        158.32GiB
    Free (estimated):            286.36GiB      (min: 286.36GiB)
    Free (statfs, df):           286.36GiB
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:              108.31MiB      (used: 0.00B)
    Multiple profiles:                  no

                  Data      Metadata  System                              
Id Path           single    single    single   Unallocated Total     Slack
-- -------------- --------- --------- -------- ----------- --------- -----
 1 /dev/nvme0n1p1 332.01GiB   3.01GiB  4.00MiB   112.11GiB 447.13GiB     -
-- -------------- --------- --------- -------- ----------- --------- -----
   Total          332.01GiB   3.01GiB  4.00MiB   112.11GiB 447.13GiB 0.00B
   Used           157.76GiB 580.64MiB 64.00KiB  

 

That's what I get for starting testing your suggestions before making my morning coffee. Very much appreciate your help and patience. 

Filesystem is mounted there, now see what you can recover, since the device has problems you might not be able to recover everything.

Sorry for probably being a bit slow, but how would I go forward with copying the contents of the cache drive to for example the folder "cacherec" on disk6? 

I downloaded Krusader and tried browsing as well but can't find the cache drive there so I'm guessing I'll have to use a command for this in the terminal?

 

Thanks again!

You can use midnight commander (mc on the console) then browse to /recovertemp on the left side and /mnt/disk6/share_name on the right.

Thank you so much! That is incredible. Will save what can be saved and install a new cache drive.