jay2068 Posted March 10 Share Posted March 10 I had a failed 3TB seagate drive. I replaced it with a 4TB WD Red. I have done replacments before as I've had my server a long time. Normally it just rebuilds and I'm good to go. This time it said it would take 11 hours. So I stopped all my VM's and dockers to make sure it wasn't doing anything. Well I wake up this morning and I can't get to the web interface and all the dockers are back up (VM's still down). So I stopped all the dockers with: docker kill $(docker ps -q) This command started showing all the container ID's and just stopped after 19 ID's. Hasn't returned to the command line in over an hour. I also ran:/etc/rc.d/rc.php-fpm restart That was to try and restart the web interface. The shutting down failed (I assume because it was already down) but that is also stuck at starting php-fpm for over an hour. I don't know where the restore is at or what I should do. I don't want to reboot if it's still working on rebuild. It's been rebuilding since yesterday at 3 PM Any help is appreciated. Quote Link to comment
JorgeB Posted March 10 Share Posted March 10 See if you can get the syslog: cp /var/log/syslog /boot/syslog.txt Quote Link to comment
jay2068 Posted March 10 Author Share Posted March 10 36 minutes ago, JorgeB said: See if you can get the syslog: cp /var/log/syslog /boot/syslog.txt Attached is the log unraid_syslog.txt Quote Link to comment
jay2068 Posted March 10 Author Share Posted March 10 (edited) I see btfrs error and it also says it requires a reboot Mar 9 22:31:11 Tower12 kernel: BTRFS critical (device loop2): corrupt leaf: block=30805327872 slot=227 extent bytenr=30800609280 len=16384 invalid generation, have 9837406843266202086 expect (0, 1967407] Mar 9 22:31:11 Tower12 kernel: BTRFS info (device loop2): leaf 30805327872 gen 1967406 total ptrs 249 free space 221 owner 2 Mar 9 22:31:11 Tower12 kernel: item 0 key (30796890112 169 0) itemoff 16250 itemsize 33 Mar 9 22:31:11 Tower12 kernel: extent refs 1 gen 1843974 flags 2 Mar 9 22:31:11 Tower12 kernel: ref#0: tree block backref root 13899 Mar 9 22:31:11 Tower12 kernel: item 1 key (30796906496 169 0) itemoff 16190 itemsize 60 and Mar 9 22:31:11 Tower12 kernel: ref#0: tree block backref root 7 Mar 9 22:31:11 Tower12 kernel: BTRFS error (device loop2): block=30805327872 write time tree block corruption detected Mar 9 22:31:13 Tower12 kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2377: errno=-5 IO failure (Error while writing out transaction) Mar 9 22:31:13 Tower12 kernel: BTRFS info (device loop2): forced readonly Mar 9 22:31:13 Tower12 kernel: BTRFS warning (device loop2): Skipping commit of aborted transaction. Mar 9 22:31:13 Tower12 kernel: BTRFS: error (device loop2) in cleanup_transaction:1942: errno=-5 IO failure Mar 9 22:31:13 Tower12 kernel: BTRFS warning (device loop2): Skipping commit of aborted transaction. Mar 9 22:31:13 Tower12 kernel: BTRFS: error (device loop2) in cleanup_transaction:1942: errno=-5 IO failure Mar 10 00:20:49 Tower12 kernel: Fixing recursive fault but reboot is needed! I think I should reboot and see where I am at. Edited March 10 by jay2068 Quote Link to comment
Solution JorgeB Posted March 10 Solution Share Posted March 10 Mar 9 22:42:41 Tower12 kernel: md: sync done. time=36575sec Mar 9 22:42:41 Tower12 kernel: md: recovery thread: exit status: 0 Rebuild finished. btrfs errors are about the docker image, delete and re-create, and try switching to ipvlan (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)), this might have been the source of the main issues. Quote Link to comment
jay2068 Posted March 10 Author Share Posted March 10 5 minutes ago, JorgeB said: Mar 9 22:42:41 Tower12 kernel: md: sync done. time=36575sec Mar 9 22:42:41 Tower12 kernel: md: recovery thread: exit status: 0 Rebuild finished. btrfs errors are about the docker image, delete and re-create, and try switching to ipvlan (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)), this might have been the source of the main issues. Ok I'll try a reboot and report back. Quote Link to comment
jay2068 Posted March 10 Author Share Posted March 10 (edited) Everyhting is back working. I modified the docker network type also. Edited March 10 by jay2068 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.