espenandre Posted February 2, 2015 Share Posted February 2, 2015 Hello, I've been trying to figure out what is wrong, but I'm a noob on Linux and hdds and stuff, so I haven't come far.. Any help is appreciated. The problem is that I'm getting a message of "read-only file system". This link is a zip containing the results of "smartctl -t short" on all the drives, and the syslog: https://onedrive.live.com/redir?resid=F442DB094E7B6625!59210&authkey=!AG4DcdHRJ1LpuTw&ithint=file%2czip Starting smartctl -t long now (in parallel on all disks) Quote Link to comment
Squid Posted February 2, 2015 Share Posted February 2, 2015 Disk 2 has been mounted as read-only because the file system is corrupted. You should run reiserfsck as per this: http://lime-technology.com/wiki/index.php/Check_Disk_Filesystems The smart report on disk 2 looks ok... Quote Link to comment
espenandre Posted February 3, 2015 Author Share Posted February 3, 2015 Thank you for the quick and helpful reply, I will take a look at it when I get back from work later today Quote Link to comment
espenandre Posted February 8, 2015 Author Share Posted February 8, 2015 After starting the array in Maintenance mode, this is the output from the reiserfsck-command on disk 2: root@Bart:~# reiserfsck --check /dev/sdb reiserfsck 3.6.24 Will read-only check consistency of the filesystem on /dev/sdb Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes reiserfs_open: the reiserfs superblock cannot be found on /dev/sdb. Failed to open the filesystem. If the partition table has not been changed, and the partition is valid and it really contains a reiserfs partition, then the superblock is corrupted and you need to run this utility with --rebuild-sb. Should I run --rebuild-sb...? Edit: Disregard this post, I didn't read through the whole page before starting the process. I substituted /dev/sdb with /dev/md2, and the check is running now. Quote Link to comment
Squid Posted February 8, 2015 Share Posted February 8, 2015 DON'T You're running reiserfsck wrong. follow these directions: http://lime-technology.com/wiki/index.php/Check_Disk_Filesystems Quote Link to comment
espenandre Posted February 8, 2015 Author Share Posted February 8, 2015 Yes, I just realized that. Running the proper reiserfsck now. Quote Link to comment
espenandre Posted February 8, 2015 Author Share Posted February 8, 2015 This is the result from reiserfsck --check: reiserfsck --check /dev/md2 reiserfsck 3.6.24 Will read-only check consistency of the filesystem on /dev/md2 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Sun Feb 8 20:55:03 2015 ########### Replaying journal: Done. Reiserfs journal '/dev/md2' in blocks [18..8211]: 0 transactions replayed Checking internal tree.. \/ 20 (of 23// 34 (of 156// 85 (of 85-bad_path: The right delimiting key [15967 15968 0x13637001 IND (1)] of the node (701861318) must be greater than the last (0) element's key [15970 10733 0x13243001 IND (1)] within the node. / 35 (of 156\/ 2 (of 170/block 237191890: The level of the node (45525) is not correct, (1) expected the problem in the internal node occured (237191890), / 36 (of 156-block 237293190: The level of the node (38879) is not correct, (2) expected the problem in the internal node occured (237293190)finished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 3 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Sun Feb 8 21:48:59 2015 ########### Quote Link to comment
Squid Posted February 8, 2015 Share Posted February 8, 2015 ok Run it with --rebuild-tree instead of --check. It will take a while to complete Quote Link to comment
espenandre Posted February 9, 2015 Author Share Posted February 9, 2015 That seems to have worked, but now I'm getting an error on the cache disk (btrfs): Feb 9 06:52:06 Bart emhttp: shcmd (240): mkdir -p /mnt/cache Feb 9 06:52:06 Bart emhttp: shcmd (241): set -o pipefail ; mount -t btrfs -o noatime,nodiratime /dev/sdf1 /mnt/cache |& logger Feb 9 06:52:06 Bart kernel: BTRFS info (device sdf1): disk space caching is enabled Feb 9 06:52:06 Bart emhttp: shcmd (242): btrfs filesystem resize max /mnt/cache |& logger Feb 9 06:52:06 Bart logger: Resize '/mnt/cache' of 'max' Feb 9 06:52:06 Bart emhttp: shcmd (243): sync Feb 9 06:52:06 Bart kernel: BTRFS: new size for /dev/sdf1 is 500106743808 Feb 9 06:52:07 Bart emhttp: shcmd (244): mkdir /mnt/user0 Feb 9 06:52:07 Bart emhttp: shcmd (245): /usr/local/sbin/shfs /mnt/user0 -disks 14 -o noatime,big_writes,allow_other |& logger Feb 9 06:52:07 Bart emhttp: shcmd (246): mkdir /mnt/user Feb 9 06:52:07 Bart emhttp: shcmd (247): /usr/local/sbin/shfs /mnt/user -disks 15 2048000000 -o noatime,big_writes,allow_other -o remember=0 |& logger Feb 9 06:52:07 Bart emhttp: shcmd (248): crontab -c /etc/cron.d - <<< "# Generated mover schedule:#01240 3 * * * /usr/local/sbin/mover |& logger" Feb 9 06:52:07 Bart emhttp: shcmd (249): /usr/local/sbin/emhttp_event disks_mounted Feb 9 06:52:07 Bart emhttp_event: disks_mounted Feb 9 06:52:07 Bart logger: Starting docker.io Feb 9 06:52:09 Bart kernel: BTRFS info (device loop8): disk space caching is enabled Feb 9 06:52:09 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9158, flush 0, corrupt 0, gen 0 Feb 9 06:52:10 Bart kernel: BTRFS: checking UUID tree Feb 9 06:52:10 Bart logger: Resize '/var/lib/docker' of 'max' Feb 9 06:52:10 Bart logger: starting docker ... Feb 9 06:52:10 Bart kernel: BTRFS: new size for /dev/loop8 is 10737418240 Feb 9 06:52:19 Bart kernel: device vethd6cb entered promiscuous mode Feb 9 06:52:19 Bart avahi-daemon[17082]: Withdrawing workstation service for veth7db0. Feb 9 06:52:19 Bart kernel: docker0: port 1(vethd6cb) entered forwarding state Feb 9 06:52:19 Bart kernel: docker0: port 1(vethd6cb) entered forwarding state Feb 9 06:52:19 Bart logger: sabnzbd: started succesfully! Feb 9 06:52:19 Bart kernel: device veth802b entered promiscuous mode Feb 9 06:52:19 Bart kernel: docker0: port 2(veth802b) entered forwarding state Feb 9 06:52:19 Bart kernel: docker0: port 2(veth802b) entered forwarding state Feb 9 06:52:19 Bart avahi-daemon[17082]: Withdrawing workstation service for vethacae. Feb 9 06:52:20 Bart logger: couchpotato: started succesfully! Feb 9 06:52:23 Bart kernel: btrfs_readpage_end_io_hook: 9 callbacks suppressed Feb 9 06:52:23 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 445890560 csum 2566472073 expected csum 1230784438 Feb 9 06:52:23 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 445890560 csum 2566472073 expected csum 1230784438 Feb 9 06:52:23 Bart kernel: btrfs_dev_stat_print_on_error: 5 callbacks suppressed Feb 9 06:52:23 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9159, flush 0, corrupt 0, gen 0 Feb 9 06:52:23 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 445874176 csum 3800195258 expected csum 582823009 Feb 9 06:52:23 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 445874176 csum 3800195258 expected csum 582823009 Feb 9 06:52:23 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9160, flush 0, corrupt 0, gen 0 Feb 9 06:52:23 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 1519616000 csum 2566472073 expected csum 582823009 Feb 9 06:52:23 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 1519616000 csum 2566472073 expected csum 582823009 Feb 9 06:52:23 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9161, flush 0, corrupt 0, gen 0 Feb 9 06:52:23 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 445874176 csum 3800195258 expected csum 582823009 Feb 9 06:52:23 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9162, flush 0, corrupt 0, gen 0 Feb 9 06:52:23 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 1519616000 csum 2566472073 expected csum 582823009 Feb 9 06:52:23 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9163, flush 0, corrupt 0, gen 0 Feb 9 06:52:24 Bart logger: CrashPlan: started succesfully! Feb 9 06:52:25 Bart kernel: device veth493f entered promiscuous mode Feb 9 06:52:25 Bart avahi-daemon[17082]: Withdrawing workstation service for veth7bcf. Feb 9 06:52:25 Bart kernel: docker0: port 3(veth493f) entered forwarding state Feb 9 06:52:25 Bart kernel: docker0: port 3(veth493f) entered forwarding state Feb 9 06:52:25 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 445923328 csum 2566472073 expected csum 2113704584 Feb 9 06:52:25 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 445923328 csum 2566472073 expected csum 2113704584 Feb 9 06:52:25 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9164, flush 0, corrupt 0, gen 0 Feb 9 06:52:25 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9165, flush 0, corrupt 0, gen 0 Feb 9 06:52:25 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9166, flush 0, corrupt 0, gen 0 Feb 9 06:52:25 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9167, flush 0, corrupt 0, gen 0 Feb 9 06:52:25 Bart kernel: ------------[ cut here ]------------ Feb 9 06:52:25 Bart kernel: WARNING: CPU: 0 PID: 17372 at fs/btrfs/super.c:259 __btrfs_abort_transaction+0x4b/0xfb() Feb 9 06:52:25 Bart kernel: BTRFS: Transaction aborted (error -5) Feb 9 06:52:25 Bart kernel: Modules linked in: md_mod veth xt_nat ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 nf_nat iptable_filter ip_tables k8temp forcedeth ahci libahci sata_sil asus_atk0110 [last unloaded: md_mod] Feb 9 06:52:25 Bart kernel: CPU: 0 PID: 17372 Comm: kworker/u8:3 Tainted: G W 3.16.3-unRAID #3 Feb 9 06:52:25 Bart kernel: Hardware name: System manufacturer System Product Name/M3N78, BIOS 0504 10/15/2008 Feb 9 06:52:25 Bart kernel: Workqueue: btrfs-extent-refs btrfs_extent_refs_helper Feb 9 06:52:25 Bart kernel: 0000000000000000 ffff88016d91ba70 ffffffff815d9532 ffff88016d91bab8 Feb 9 06:52:25 Bart kernel: ffff88016d91baa8 ffffffff81040687 ffffffff81294a12 00000000fffffffb Feb 9 06:52:25 Bart kernel: ffff88018de6e000 ffff88013b317d20 ffffffff8161efa0 ffff88016d91bb08 Feb 9 06:52:25 Bart kernel: Call Trace: Feb 9 06:52:25 Bart kernel: [<ffffffff815d9532>] dump_stack+0x45/0x56 Feb 9 06:52:25 Bart kernel: [<ffffffff81040687>] warn_slowpath_common+0x75/0x8e Feb 9 06:52:25 Bart kernel: [<ffffffff81294a12>] ? __btrfs_abort_transaction+0x4b/0xfb Feb 9 06:52:25 Bart kernel: [<ffffffff810406e7>] warn_slowpath_fmt+0x47/0x49 Feb 9 06:52:25 Bart kernel: [<ffffffff812d0409>] ? set_extent_dirty+0x18/0x1a Feb 9 06:52:25 Bart kernel: [<ffffffff81294a12>] __btrfs_abort_transaction+0x4b/0xfb Feb 9 06:52:25 Bart kernel: [<ffffffff812a43e4>] __btrfs_free_extent+0x878/0x88c Feb 9 06:52:25 Bart kernel: [<ffffffff812f34f1>] ? find_ref_head+0x45/0x62 Feb 9 06:52:25 Bart kernel: [<ffffffff812a9480>] __btrfs_run_delayed_refs+0xab9/0xcc5 Feb 9 06:52:25 Bart kernel: [<ffffffff812ab26c>] btrfs_run_delayed_refs+0x69/0x21c Feb 9 06:52:25 Bart kernel: [<ffffffff812ab456>] delayed_ref_async_start+0x37/0x76 Feb 9 06:52:25 Bart kernel: [<ffffffff812dedb3>] normal_work_helper+0xbb/0x16f Feb 9 06:52:25 Bart kernel: [<ffffffff812df040>] btrfs_extent_refs_helper+0xd/0xf Feb 9 06:52:25 Bart kernel: [<ffffffff81052e85>] process_one_work+0x20d/0x332 Feb 9 06:52:25 Bart kernel: [<ffffffff81053b39>] worker_thread+0x392/0x507 Feb 9 06:52:25 Bart kernel: [<ffffffff810537a7>] ? cancel_delayed_work_sync+0x10/0x10 Feb 9 06:52:25 Bart kernel: [<ffffffff810588ba>] kthread+0xd6/0xde Feb 9 06:52:25 Bart kernel: [<ffffffff810587e4>] ? kthread_create_on_node+0x168/0x168 Feb 9 06:52:25 Bart kernel: [<ffffffff815df3bc>] ret_from_fork+0x7c/0xb0 Feb 9 06:52:25 Bart kernel: [<ffffffff810587e4>] ? kthread_create_on_node+0x168/0x168 Feb 9 06:52:25 Bart kernel: ---[ end trace 22e67f7ef0ad5e12 ]--- Feb 9 06:52:25 Bart kernel: BTRFS: error (device loop8) in __btrfs_free_extent:5965: errno=-5 IO failure Feb 9 06:52:25 Bart kernel: BTRFS info (device loop8): forced readonly Feb 9 06:52:25 Bart kernel: BTRFS: error (device loop8) in btrfs_run_delayed_refs:2806: errno=-5 IO failure Feb 9 06:52:27 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9168, flush 0, corrupt 0, gen 0 Feb 9 06:52:27 Bart kernel: BTRFS info (device loop8): no csum found for inode 18763 start 4096 Feb 9 06:52:27 Bart logger: SickBeard2: started succesfully! Feb 9 06:52:27 Bart kernel: BTRFS info (device loop8): no csum found for inode 18763 start 0 Feb 9 06:52:27 Bart pms: Starting Plex... Feb 9 06:52:27 Bart su[17660]: Successful su for unraid-plex by root Feb 9 06:52:27 Bart su[17660]: + root:unraid-plex Feb 9 06:52:27 Bart avahi-daemon[17082]: Withdrawing workstation service for veth802b. Feb 9 06:52:27 Bart kernel: docker0: port 2(veth802b) entered disabled state Feb 9 06:52:27 Bart kernel: device veth802b left promiscuous mode Feb 9 06:52:27 Bart kernel: docker0: port 2(veth802b) entered disabled state Feb 9 06:52:29 Bart pms: Plex Media Server IS running Feb 9 06:52:29 Bart emhttp: shcmd (250): :>/etc/samba/smb-shares.conf Feb 9 06:52:29 Bart avahi-daemon[17082]: Files changed, reloading. Feb 9 06:52:29 Bart emhttp: Restart SMB... Feb 9 06:52:29 Bart emhttp: shcmd (251): killall -HUP smbd Feb 9 06:52:29 Bart emhttp: shcmd (252): cp /etc/avahi/services/smb.service- /etc/avahi/services/smb.service Feb 9 06:52:29 Bart avahi-daemon[17082]: Files changed, reloading. Feb 9 06:52:29 Bart avahi-daemon[17082]: Service group file /services/smb.service changed, reloading. Feb 9 06:52:29 Bart emhttp: shcmd (253): pidof rpc.mountd &> /dev/null Feb 9 06:52:29 Bart emhttp: shcmd (254): /etc/rc.d/rc.atalk status Feb 9 06:52:29 Bart emhttp: shcmd (255): /usr/local/sbin/emhttp_event svcs_restarted Feb 9 06:52:29 Bart emhttp_event: svcs_restarted Feb 9 06:52:29 Bart emhttp: shcmd (256): /usr/local/sbin/emhttp_event started Feb 9 06:52:29 Bart emhttp_event: started Feb 9 06:52:30 Bart avahi-daemon[17082]: Service "Bart" (/services/smb.service) successfully established. Feb 9 06:52:33 Bart kernel: docker0: port 3(veth493f) entered disabled state Feb 9 06:52:33 Bart avahi-daemon[17082]: Withdrawing workstation service for veth493f. Feb 9 06:52:33 Bart kernel: device veth493f left promiscuous mode Feb 9 06:52:33 Bart kernel: docker0: port 3(veth493f) entered disabled state Feb 9 06:52:34 Bart kernel: docker0: port 1(vethd6cb) entered forwarding state Feb 9 06:53:07 Bart php: /usr/bin/docker start sickbeard Feb 9 06:53:07 Bart php: Docker is on the cache disk, can only start 1 of 4 containers.. Quote Link to comment
BRiT Posted February 9, 2015 Share Posted February 9, 2015 What does the following command show for /var/lib/docker and /mnt/cache? df -h Your docker.img might be full and needs to be recreated, or your cache drive might have issues. Its hard for me to tell from the log which BTRFs is having the issue. Quote Link to comment
espenandre Posted February 9, 2015 Author Share Posted February 9, 2015 I will try that command as soon as I get home from work. [...] Your docker.img might be full and needs to be recreated, or your cache drive might have issues. Its hard for me to tell from the log which BTRFs is having the issue. I only have one BTRFS-disk, the cache disk. Quote Link to comment
BRiT Posted February 9, 2015 Share Posted February 9, 2015 I will try that command as soon as I get home from work. [...] Your docker.img might be full and needs to be recreated, or your cache drive might have issues. Its hard for me to tell from the log which BTRFs is having the issue. I only have one BTRFS-disk, the cache disk. Not exactly true. Inside the docker.img itself is a BTRFS filesystem. Quote Link to comment
espenandre Posted February 9, 2015 Author Share Posted February 9, 2015 What does the following command show for /var/lib/docker and /mnt/cache? df -h Your docker.img might be full and needs to be recreated, or your cache drive might have issues. Its hard for me to tell from the log which BTRFs is having the issue. root@Bart:~# df -h /var/lib/docker/ Filesystem Size Used Avail Use% Mounted on /dev/loop8 10G 2.9G 5.5G 34% /var/lib/docker root@Bart:~# df -h /mnt/cache/ Filesystem Size Used Avail Use% Mounted on /dev/sdf1 466G 69G 396G 15% /mnt/cache Thanks for helping me! Edit: Here is a link to the current syslog, if it helps: https://www.dropbox.com/s/u5hldj9zbnvzo8w/syslog?dl=0 Quote Link to comment
BRiT Posted February 9, 2015 Share Posted February 9, 2015 Alright, based on that output your issue isnt caused from a full filesystem. Your cache drive and docker.img have plenty of free space. At least they report as such. I dont exactly know what steps to take next on trying to fix the btrfs issues. There is a filesystem check profram. Hopefully jonp or eric or LimeTech or one of the moderators can step in to help further. Quote Link to comment
espenandre Posted February 9, 2015 Author Share Posted February 9, 2015 If I replace the cache disk (it's getting old anyways, and I've got a spare 1.5TB WD Green laying around), could I then try to copy all the content to the new disk, or would that just transfer the problems to the new disk as well? Optionally, I could just copy the configurations on my Docker containers (config files of i.e. sickbeard etc), to save some setup-time on the new disk. Quote Link to comment
espenandre Posted February 10, 2015 Author Share Posted February 10, 2015 I did try a scrub on the cache disk, this is the output: root@Bart:~# btrfs scrub start /mnt/cache scrub started on /mnt/cache, fsid 989a9240-778e-4c53-bf50-e92b02537dcc (pid=15930) root@Bart:~# ERROR: There are uncorrectable errors. root@Bart:~# I did start a new Docker image, as you mentioned that it is possible that is the culprit, and added the containers again over the existing docker folders (keeping settings and stuff intact, but downloading and installing the container fresh). At least the docker containers are starting up and seemingly are working. For now, at least. This is the latest syslog, should it be of interest: https://www.dropbox.com/s/iwzqq5hwylsqiul/15-10-02_syslog?dl=0 Quote Link to comment
fespinoza831 Posted October 10, 2022 Share Posted October 10, 2022 Hello, eveery other day or so some of my containers will stop working because my cause drive isset as a read only file system. However when this happens. I reboot the server and everything is back to normal. Not quite sure what is going on here. Bad drive? Can someone point me in the right direction. What logs may be needed for help? Quote Link to comment
ChatNoir Posted October 10, 2022 Share Posted October 10, 2022 4 minutes ago, fespinoza831 said: Hello, eveery other day or so some of my containers will stop working because my cause drive isset as a read only file system. However when this happens. I reboot the server and everything is back to normal. Not quite sure what is going on here. Bad drive? Can someone point me in the right direction. What logs may be needed for help? It's a 7 years old thread. You should probably start a new one and provide your diagnostics. Quote Link to comment
fespinoza831 Posted October 10, 2022 Share Posted October 10, 2022 Will do, Sorry about that was searching and didn't notice when it was posted. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.