November 13, 20232 yr Hello everyone, today during some heavy load due to some mass-uploading to Immich, my Cache Pool became read only. These were the steps I did after I realized there is something wrong: Stopped all Docker containers gracefully Disabled Docker Stopped the Array Restarted the System Started the array (Cache Pool still "Unmountable: No file system") Did some steps to recover files:mkdir /temp and mount -o rescue=all,ro /dev/mapper/nvme0n1p1 /temp worked for me cp -avr /temp /mnt/disk5/backup_unmountable Right now I'm waiting for the copy / backup of my cache drive to finish. The Cache Pool is btrfs encrypted and files are accessible for now. So far, this is what happened.. But now I have two questions: Is there a way to find out why this happened? Can I fix the filesystem without reformatting? Should I use btrfs check --repair? Diagnostics are attached. Thanks and kind regards serverpc-diagnostics-20231113-1729.zip
November 13, 20232 yr Community Expert Solution If the log tree is the only problem, this may help, with the array started type: btrfs rescue zero-log /dev/mapper/nvme0n1p1 Then re-start the array
November 13, 20232 yr Author I also did this for the other disk (nvme1n1p1) It worked, thank you!!! 🎉 Edited November 13, 20232 yr by 3dee
November 14, 20232 yr Author A couple of hours later it happened again This time I was collecting the logs right after stopping the Array. I had issues stopping Docker Containers and stopping the Array (device busy but no lsof results). I was able to unmount manually via umount -l since -f didn't work. The mover was doing some heavy Cache Pool I/O when the first BTRFS error occurred, just like the first time when this happened. Nov 13 22:49:28 ServerPC move: file: //..g/... ### [PREVIOUS LINE REPEATED 4 TIMES] ### Nov 13 22:49:30 ServerPC move: file: //..p/... ### [PREVIOUS LINE REPEATED 6 TIMES] ### Nov 13 22:49:34 ServerPC move: skip: /mnt/nvme_pool/immich_gallery/thumbs/15c8cb25-0e87-4575-b952-a140ec2063fa/ee/16/ee169c32-9514-4527-81a8-45c1d6b1e199.jpeg Nov 13 22:49:34 ServerPC move: file: //..p/... ### [PREVIOUS LINE REPEATED 2 TIMES] ### Nov 13 22:49:35 ServerPC move: file: //..g/... Nov 13 22:49:36 ServerPC move: file: //..p/... ### [PREVIOUS LINE REPEATED 10 TIMES] ### Nov 13 22:49:41 ServerPC move: file: //..g/... Nov 13 22:49:41 ServerPC move: file: //..p/... ### [PREVIOUS LINE REPEATED 14 TIMES] ### Nov 13 22:49:48 ServerPC kernel: BTRFS error (device dm-8): parent transid verify failed on 484232986624 wanted 18446612690567976080 found 180123 ### [PREVIOUS LINE REPEATED 1 TIMES] ### Nov 13 22:49:48 ServerPC kernel: BTRFS: error (device dm-8: state A) in do_free_extent_accounting:2852: errno=-5 IO failure Nov 13 22:49:48 ServerPC kernel: BTRFS info (device dm-8: state EA): forced readonly Nov 13 22:49:48 ServerPC kernel: BTRFS: error (device dm-8: state EA) in btrfs_run_delayed_refs:2157: errno=-5 IO failure Nov 13 22:49:48 ServerPC rsyslogd: file '/mnt/user/syslog/syslog-192.168.1.10.log'[10] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: Read-only file system [v8.2102.0 try https://www.rsyslog.com/e/2027 ] Nov 13 22:49:48 ServerPC rsyslogd: action 'action-3-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2102.0 try https://www.rsyslog.com/e/2027 ] Nov 13 22:49:48 ServerPC rsyslogd: file '/mnt/user/syslog/syslog-192.168.1.10.log'[10] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: Read-only file system [v8.2102.0 try https://www.rsyslog.com/e/2027 ] Nov 13 22:49:48 ServerPC rsyslogd: action 'action-3-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2102.0 try https://www.rsyslog.com/e/2027 ] Nov 13 22:49:48 ServerPC rsyslogd: file '/mnt/user/syslog/syslog-192.168.1.10.log'[10] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: Read-only file system [v8.2102.0 try https://www.rsyslog.com/e/2027 ] Nov 13 22:49:48 ServerPC rsyslogd: action 'action-3-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2102.0 try https://www.rsyslog.com/e/2027 ] Nov 13 22:49:48 ServerPC rsyslogd: file '/mnt/user/syslog/syslog-192.168.1.10.log'[10] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: Read-only file system [v8.2102.0 try https://www.rsyslog.com/e/2027 ] Nov 13 22:49:48 ServerPC rsyslogd: action 'action-3-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2102.0 try https://www.rsyslog.com/e/2027 ] Can you help me finding out what causes this problem? New diagnostics are attached.. Edit: I will look into this: serverpc-diagnostics-20231114-0049.zip Edited November 14, 20232 yr by 3dee
November 14, 20232 yr Community Expert 56 minutes ago, 3dee said: A couple of hours later it happened agai The pool has other issues, suggest to backup and re-format.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.