espenandre

I did try a scrub on the cache disk, this is the output: root@Bart:~# btrfs scrub start /mnt/cache scrub started on /mnt/cache, fsid 989a9240-778e-4c53-bf50-e92b02537dcc (pid=15930) root@Bart:~# ERROR: There are uncorrectable errors. root@Bart:~# I did start a new Docker image, as you mentioned that it is possible that is the culprit, and added the containers again over the existing docker folders (keeping settings and stuff intact, but downloading and installing the container fresh). At least the docker containers are starting up and seemingly are working. For now, at least. This is the latest syslog, should it be of interest: https://www.dropbox.com/s/iwzqq5hwylsqiul/15-10-02_syslog?dl=0

If I replace the cache disk (it's getting old anyways, and I've got a spare 1.5TB WD Green laying around), could I then try to copy all the content to the new disk, or would that just transfer the problems to the new disk as well? Optionally, I could just copy the configurations on my Docker containers (config files of i.e. sickbeard etc), to save some setup-time on the new disk.

root@Bart:~# df -h /var/lib/docker/ Filesystem Size Used Avail Use% Mounted on /dev/loop8 10G 2.9G 5.5G 34% /var/lib/docker root@Bart:~# df -h /mnt/cache/ Filesystem Size Used Avail Use% Mounted on /dev/sdf1 466G 69G 396G 15% /mnt/cache Thanks for helping me! Edit: Here is a link to the current syslog, if it helps: https://www.dropbox.com/s/u5hldj9zbnvzo8w/syslog?dl=0

I will try that command as soon as I get home from work. I only have one BTRFS-disk, the cache disk.

That seems to have worked, but now I'm getting an error on the cache disk (btrfs): Feb 9 06:52:06 Bart emhttp: shcmd (240): mkdir -p /mnt/cache Feb 9 06:52:06 Bart emhttp: shcmd (241): set -o pipefail ; mount -t btrfs -o noatime,nodiratime /dev/sdf1 /mnt/cache |& logger Feb 9 06:52:06 Bart kernel: BTRFS info (device sdf1): disk space caching is enabled Feb 9 06:52:06 Bart emhttp: shcmd (242): btrfs filesystem resize max /mnt/cache |& logger Feb 9 06:52:06 Bart logger: Resize '/mnt/cache' of 'max' Feb 9 06:52:06 Bart emhttp: shcmd (243): sync Feb 9 06:52:06 Bart kernel: BTRFS: new size for /dev/sdf1 is 500106743808 Feb 9 06:52:07 Bart emhttp: shcmd (244): mkdir /mnt/user0 Feb 9 06:52:07 Bart emhttp: shcmd (245): /usr/local/sbin/shfs /mnt/user0 -disks 14 -o noatime,big_writes,allow_other |& logger Feb 9 06:52:07 Bart emhttp: shcmd (246): mkdir /mnt/user Feb 9 06:52:07 Bart emhttp: shcmd (247): /usr/local/sbin/shfs /mnt/user -disks 15 2048000000 -o noatime,big_writes,allow_other -o remember=0 |& logger Feb 9 06:52:07 Bart emhttp: shcmd (248): crontab -c /etc/cron.d - <<< "# Generated mover schedule:#01240 3 * * * /usr/local/sbin/mover |& logger" Feb 9 06:52:07 Bart emhttp: shcmd (249): /usr/local/sbin/emhttp_event disks_mounted Feb 9 06:52:07 Bart emhttp_event: disks_mounted Feb 9 06:52:07 Bart logger: Starting docker.io Feb 9 06:52:09 Bart kernel: BTRFS info (device loop8): disk space caching is enabled Feb 9 06:52:09 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9158, flush 0, corrupt 0, gen 0 Feb 9 06:52:10 Bart kernel: BTRFS: checking UUID tree Feb 9 06:52:10 Bart logger: Resize '/var/lib/docker' of 'max' Feb 9 06:52:10 Bart logger: starting docker ... Feb 9 06:52:10 Bart kernel: BTRFS: new size for /dev/loop8 is 10737418240 Feb 9 06:52:19 Bart kernel: device vethd6cb entered promiscuous mode Feb 9 06:52:19 Bart avahi-daemon[17082]: Withdrawing workstation service for veth7db0. Feb 9 06:52:19 Bart kernel: docker0: port 1(vethd6cb) entered forwarding state Feb 9 06:52:19 Bart kernel: docker0: port 1(vethd6cb) entered forwarding state Feb 9 06:52:19 Bart logger: sabnzbd: started succesfully! Feb 9 06:52:19 Bart kernel: device veth802b entered promiscuous mode Feb 9 06:52:19 Bart kernel: docker0: port 2(veth802b) entered forwarding state Feb 9 06:52:19 Bart kernel: docker0: port 2(veth802b) entered forwarding state Feb 9 06:52:19 Bart avahi-daemon[17082]: Withdrawing workstation service for vethacae. Feb 9 06:52:20 Bart logger: couchpotato: started succesfully! Feb 9 06:52:23 Bart kernel: btrfs_readpage_end_io_hook: 9 callbacks suppressed Feb 9 06:52:23 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 445890560 csum 2566472073 expected csum 1230784438 Feb 9 06:52:23 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 445890560 csum 2566472073 expected csum 1230784438 Feb 9 06:52:23 Bart kernel: btrfs_dev_stat_print_on_error: 5 callbacks suppressed Feb 9 06:52:23 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9159, flush 0, corrupt 0, gen 0 Feb 9 06:52:23 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 445874176 csum 3800195258 expected csum 582823009 Feb 9 06:52:23 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 445874176 csum 3800195258 expected csum 582823009 Feb 9 06:52:23 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9160, flush 0, corrupt 0, gen 0 Feb 9 06:52:23 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 1519616000 csum 2566472073 expected csum 582823009 Feb 9 06:52:23 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 1519616000 csum 2566472073 expected csum 582823009 Feb 9 06:52:23 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9161, flush 0, corrupt 0, gen 0 Feb 9 06:52:23 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 445874176 csum 3800195258 expected csum 582823009 Feb 9 06:52:23 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9162, flush 0, corrupt 0, gen 0 Feb 9 06:52:23 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 1519616000 csum 2566472073 expected csum 582823009 Feb 9 06:52:23 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9163, flush 0, corrupt 0, gen 0 Feb 9 06:52:24 Bart logger: CrashPlan: started succesfully! Feb 9 06:52:25 Bart kernel: device veth493f entered promiscuous mode Feb 9 06:52:25 Bart avahi-daemon[17082]: Withdrawing workstation service for veth7bcf. Feb 9 06:52:25 Bart kernel: docker0: port 3(veth493f) entered forwarding state Feb 9 06:52:25 Bart kernel: docker0: port 3(veth493f) entered forwarding state Feb 9 06:52:25 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 445923328 csum 2566472073 expected csum 2113704584 Feb 9 06:52:25 Bart kernel: BTRFS info (device sdf1): csum failed ino 162080 off 445923328 csum 2566472073 expected csum 2113704584 Feb 9 06:52:25 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9164, flush 0, corrupt 0, gen 0 Feb 9 06:52:25 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9165, flush 0, corrupt 0, gen 0 Feb 9 06:52:25 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9166, flush 0, corrupt 0, gen 0 Feb 9 06:52:25 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9167, flush 0, corrupt 0, gen 0 Feb 9 06:52:25 Bart kernel: ------------[ cut here ]------------ Feb 9 06:52:25 Bart kernel: WARNING: CPU: 0 PID: 17372 at fs/btrfs/super.c:259 __btrfs_abort_transaction+0x4b/0xfb() Feb 9 06:52:25 Bart kernel: BTRFS: Transaction aborted (error -5) Feb 9 06:52:25 Bart kernel: Modules linked in: md_mod veth xt_nat ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 nf_nat iptable_filter ip_tables k8temp forcedeth ahci libahci sata_sil asus_atk0110 [last unloaded: md_mod] Feb 9 06:52:25 Bart kernel: CPU: 0 PID: 17372 Comm: kworker/u8:3 Tainted: G W 3.16.3-unRAID #3 Feb 9 06:52:25 Bart kernel: Hardware name: System manufacturer System Product Name/M3N78, BIOS 0504 10/15/2008 Feb 9 06:52:25 Bart kernel: Workqueue: btrfs-extent-refs btrfs_extent_refs_helper Feb 9 06:52:25 Bart kernel: 0000000000000000 ffff88016d91ba70 ffffffff815d9532 ffff88016d91bab8 Feb 9 06:52:25 Bart kernel: ffff88016d91baa8 ffffffff81040687 ffffffff81294a12 00000000fffffffb Feb 9 06:52:25 Bart kernel: ffff88018de6e000 ffff88013b317d20 ffffffff8161efa0 ffff88016d91bb08 Feb 9 06:52:25 Bart kernel: Call Trace: Feb 9 06:52:25 Bart kernel: [<ffffffff815d9532>] dump_stack+0x45/0x56 Feb 9 06:52:25 Bart kernel: [<ffffffff81040687>] warn_slowpath_common+0x75/0x8e Feb 9 06:52:25 Bart kernel: [<ffffffff81294a12>] ? __btrfs_abort_transaction+0x4b/0xfb Feb 9 06:52:25 Bart kernel: [<ffffffff810406e7>] warn_slowpath_fmt+0x47/0x49 Feb 9 06:52:25 Bart kernel: [<ffffffff812d0409>] ? set_extent_dirty+0x18/0x1a Feb 9 06:52:25 Bart kernel: [<ffffffff81294a12>] __btrfs_abort_transaction+0x4b/0xfb Feb 9 06:52:25 Bart kernel: [<ffffffff812a43e4>] __btrfs_free_extent+0x878/0x88c Feb 9 06:52:25 Bart kernel: [<ffffffff812f34f1>] ? find_ref_head+0x45/0x62 Feb 9 06:52:25 Bart kernel: [<ffffffff812a9480>] __btrfs_run_delayed_refs+0xab9/0xcc5 Feb 9 06:52:25 Bart kernel: [<ffffffff812ab26c>] btrfs_run_delayed_refs+0x69/0x21c Feb 9 06:52:25 Bart kernel: [<ffffffff812ab456>] delayed_ref_async_start+0x37/0x76 Feb 9 06:52:25 Bart kernel: [<ffffffff812dedb3>] normal_work_helper+0xbb/0x16f Feb 9 06:52:25 Bart kernel: [<ffffffff812df040>] btrfs_extent_refs_helper+0xd/0xf Feb 9 06:52:25 Bart kernel: [<ffffffff81052e85>] process_one_work+0x20d/0x332 Feb 9 06:52:25 Bart kernel: [<ffffffff81053b39>] worker_thread+0x392/0x507 Feb 9 06:52:25 Bart kernel: [<ffffffff810537a7>] ? cancel_delayed_work_sync+0x10/0x10 Feb 9 06:52:25 Bart kernel: [<ffffffff810588ba>] kthread+0xd6/0xde Feb 9 06:52:25 Bart kernel: [<ffffffff810587e4>] ? kthread_create_on_node+0x168/0x168 Feb 9 06:52:25 Bart kernel: [<ffffffff815df3bc>] ret_from_fork+0x7c/0xb0 Feb 9 06:52:25 Bart kernel: [<ffffffff810587e4>] ? kthread_create_on_node+0x168/0x168 Feb 9 06:52:25 Bart kernel: ---[ end trace 22e67f7ef0ad5e12 ]--- Feb 9 06:52:25 Bart kernel: BTRFS: error (device loop8) in __btrfs_free_extent:5965: errno=-5 IO failure Feb 9 06:52:25 Bart kernel: BTRFS info (device loop8): forced readonly Feb 9 06:52:25 Bart kernel: BTRFS: error (device loop8) in btrfs_run_delayed_refs:2806: errno=-5 IO failure Feb 9 06:52:27 Bart kernel: BTRFS: bdev /dev/loop8 errs: wr 0, rd 9168, flush 0, corrupt 0, gen 0 Feb 9 06:52:27 Bart kernel: BTRFS info (device loop8): no csum found for inode 18763 start 4096 Feb 9 06:52:27 Bart logger: SickBeard2: started succesfully! Feb 9 06:52:27 Bart kernel: BTRFS info (device loop8): no csum found for inode 18763 start 0 Feb 9 06:52:27 Bart pms: Starting Plex... Feb 9 06:52:27 Bart su[17660]: Successful su for unraid-plex by root Feb 9 06:52:27 Bart su[17660]: + root:unraid-plex Feb 9 06:52:27 Bart avahi-daemon[17082]: Withdrawing workstation service for veth802b. Feb 9 06:52:27 Bart kernel: docker0: port 2(veth802b) entered disabled state Feb 9 06:52:27 Bart kernel: device veth802b left promiscuous mode Feb 9 06:52:27 Bart kernel: docker0: port 2(veth802b) entered disabled state Feb 9 06:52:29 Bart pms: Plex Media Server IS running Feb 9 06:52:29 Bart emhttp: shcmd (250): :>/etc/samba/smb-shares.conf Feb 9 06:52:29 Bart avahi-daemon[17082]: Files changed, reloading. Feb 9 06:52:29 Bart emhttp: Restart SMB... Feb 9 06:52:29 Bart emhttp: shcmd (251): killall -HUP smbd Feb 9 06:52:29 Bart emhttp: shcmd (252): cp /etc/avahi/services/smb.service- /etc/avahi/services/smb.service Feb 9 06:52:29 Bart avahi-daemon[17082]: Files changed, reloading. Feb 9 06:52:29 Bart avahi-daemon[17082]: Service group file /services/smb.service changed, reloading. Feb 9 06:52:29 Bart emhttp: shcmd (253): pidof rpc.mountd &> /dev/null Feb 9 06:52:29 Bart emhttp: shcmd (254): /etc/rc.d/rc.atalk status Feb 9 06:52:29 Bart emhttp: shcmd (255): /usr/local/sbin/emhttp_event svcs_restarted Feb 9 06:52:29 Bart emhttp_event: svcs_restarted Feb 9 06:52:29 Bart emhttp: shcmd (256): /usr/local/sbin/emhttp_event started Feb 9 06:52:29 Bart emhttp_event: started Feb 9 06:52:30 Bart avahi-daemon[17082]: Service "Bart" (/services/smb.service) successfully established. Feb 9 06:52:33 Bart kernel: docker0: port 3(veth493f) entered disabled state Feb 9 06:52:33 Bart avahi-daemon[17082]: Withdrawing workstation service for veth493f. Feb 9 06:52:33 Bart kernel: device veth493f left promiscuous mode Feb 9 06:52:33 Bart kernel: docker0: port 3(veth493f) entered disabled state Feb 9 06:52:34 Bart kernel: docker0: port 1(vethd6cb) entered forwarding state Feb 9 06:53:07 Bart php: /usr/bin/docker start sickbeard Feb 9 06:53:07 Bart php: Docker is on the cache disk, can only start 1 of 4 containers..

This is the result from reiserfsck --check: reiserfsck --check /dev/md2 reiserfsck 3.6.24 Will read-only check consistency of the filesystem on /dev/md2 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Sun Feb 8 20:55:03 2015 ########### Replaying journal: Done. Reiserfs journal '/dev/md2' in blocks [18..8211]: 0 transactions replayed Checking internal tree.. \/ 20 (of 23// 34 (of 156// 85 (of 85-bad_path: The right delimiting key [15967 15968 0x13637001 IND (1)] of the node (701861318) must be greater than the last (0) element's key [15970 10733 0x13243001 IND (1)] within the node. / 35 (of 156\/ 2 (of 170/block 237191890: The level of the node (45525) is not correct, (1) expected the problem in the internal node occured (237191890), / 36 (of 156-block 237293190: The level of the node (38879) is not correct, (2) expected the problem in the internal node occured (237293190)finished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 3 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Sun Feb 8 21:48:59 2015 ###########

Yes, I just realized that. Running the proper reiserfsck now.

After starting the array in Maintenance mode, this is the output from the reiserfsck-command on disk 2: root@Bart:~# reiserfsck --check /dev/sdb reiserfsck 3.6.24 Will read-only check consistency of the filesystem on /dev/sdb Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes reiserfs_open: the reiserfs superblock cannot be found on /dev/sdb. Failed to open the filesystem. If the partition table has not been changed, and the partition is valid and it really contains a reiserfs partition, then the superblock is corrupted and you need to run this utility with --rebuild-sb. Should I run --rebuild-sb...? Edit: Disregard this post, I didn't read through the whole page before starting the process. I substituted /dev/sdb with /dev/md2, and the check is running now.

Thank you for the quick and helpful reply, I will take a look at it when I get back from work later today

Hello, I've been trying to figure out what is wrong, but I'm a noob on Linux and hdds and stuff, so I haven't come far.. Any help is appreciated. The problem is that I'm getting a message of "read-only file system". This link is a zip containing the results of "smartctl -t short" on all the drives, and the syslog: https://onedrive.live.com/redir?resid=F442DB094E7B6625!59210&authkey=!AG4DcdHRJ1LpuTw&ithint=file%2czip Starting smartctl -t long now (in parallel on all disks)