Disk is read only but shows mounted as RW (BTRFS)

[ihansen]

Hi,

 

I've been using unraid for several years now, but due to a massive disk crash not to long ago, I reinstalled the server and changed my disks to BTRFS to get snapshot and versions of files. Things have been smooth sailing, but this week my docker containers suddenly were unresponsive. I tried all the tricks i could find on the forums, but the docker daemon and containers were all loced up. By chance I noticed some errors in the log when writing to some of my shares (but not all), that stated that the write operation stopped, because it was a read only filesystem, but the mount command shows no information about this. As far as i can se, the dashboard does not show this information either.

 

My server is just now finishing a parity check after a reboot (only way to fix the docker issue) What should be the first step after that finishes? As BTRFS is quite new to me, I'm reluctant to keep poking around, and would really appreciate if someone with a better understanding then myself would lend their eyes on my logfiles and help me understand what has happened.

 

Shows filesystem as read only:

root@Enterprise:~#
root@Enterprise:~# touch /mnt/disk1 helloforums
root@Enterprise:~# touch /mnt/disk2 helloforums
touch: setting times of '/mnt/disk2': Read-only file system
root@Enterprise:~# touch /mnt/disk3 helloforums
root@Enterprise:~# touch /mnt/disk4 helloforums
root@Enterprise:~#

 

But mount does not show as read only:

root@Enterprise:~# mount
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
tmpfs on /dev/shm type tmpfs (rw)
tmpfs on /var/log type tmpfs (rw,size=128m,mode=0755)
/dev/sdb1 on /boot type vfat (rw,noatime,nodiratime,flush,dmask=77,fmask=177,shortname=mixed)
/boot/bzmodules on /lib/modules type squashfs (ro)
/boot/bzfirmware on /lib/firmware type squashfs (ro)
hugetlbfs on /hugetlbfs type hugetlbfs (rw)
overlay on /lib/modules type overlay (rw,lowerdir=/lib/modules,upperdir=/var/local/overlay/lib/modules,workdir=/var/local/overlay-work/lib/modules)
overlay on /lib/firmware type overlay (rw,lowerdir=/lib/firmware,upperdir=/var/local/overlay/lib/firmware,workdir=/var/local/overlay-work/lib/firmware)
/mnt on /mnt type none (rw,bind)
/dev/md1 on /mnt/disk1 type btrfs (rw,noatime,space_cache=v2)
/dev/md2 on /mnt/disk2 type btrfs (rw,noatime,space_cache=v2)
/dev/md3 on /mnt/disk3 type btrfs (rw,noatime,space_cache=v2)
/dev/md4 on /mnt/disk4 type btrfs (rw,noatime,space_cache=v2)
/dev/sdp1 on /mnt/cache type btrfs (rw,noatime,space_cache=v2)
/dev/sda1 on /mnt/tempdata type btrfs (rw,noatime,space_cache=v2)
shfs on /mnt/user0 type fuse.shfs (rw,nosuid,nodev,noatime,allow_other)
shfs on /mnt/user type fuse.shfs (rw,nosuid,nodev,noatime,allow_other)
/mnt/cache/system/docker/docker.img on /var/lib/docker type btrfs (rw,noatime,space_cache=v2)
/mnt/cache/system/libvirt/libvirt.img on /etc/libvirt type btrfs (rw,noatime,space_cache=v2)
gdrive_crypt: on /mnt/user/mount_rclone/gdrive_crypt type fuse.rclone (rw,nosuid,nodev,allow_other)
rclone_upload/gdrive_crypt:mount_rclone/gdrive_crypt on /mnt/user/mount_mergerfs/gdrive_crypt type fuse.mergerfs (rw,nosuid,nodev,allow_other,default_permissions)
root@Enterprise:~#

 

 

 

 

enterprise-diagnostics-20220119-1039.zip

[JorgeB]

Jan 18 20:44:37 Enterprise kernel: BTRFS info (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
Jan 18 20:44:40 Enterprise kernel: BTRFS info (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0

 

Btrfs is detecting data corruption in multiple disks, this is usually the result of RAM issues, Ryzen with RAM above max officially supported speeds is known to cause data corruption in some cases.

 

As for disk 2 there's also filesystem corruption, probably resulting from the same issues, best bet is to backup and re-format the disk, ideally after fixing the hardware problem.

[ihansen]

Wow - thanks @JorgeB for the quick reply and the pointer to the thread regarding Ryzen and RAM issues. 

 

I did not find any references to C-states in the BIOS (will google some more to identify this), but i have corrected the memory speeds now to 1866 MT/s as recommended. The BIOS was misleading me here, as it listed the setting as 1866 MHz, but it was actually referring to 1866 MT/s.

 

I have an active spare in my server, so will try to copy the data and reformat the disk. I found this witch looks like the best guide to follow for this operation.

 

Thanks again!

Edited January 19, 2022 by ihansen

[JorgeB]

25 minutes ago, ihansen said:

I found this

Since the disk still mounts you don't need to use that, you can't write to it but can still read it, so just copy the data normally using your favorite tool.