Faild disk replacment. Getting 500 Internal Server Error. Rebuilding Done? Now what do I do?

[jay2068]

I had a failed 3TB seagate drive. I replaced it with a 4TB WD Red. I have done replacments before as I've had my server a long time. Normally it just rebuilds and I'm good to go. This time it said it would take 11 hours. So I stopped all my VM's and dockers to make sure it wasn't doing anything. Well I wake up this morning and I can't get to the web interface and all the dockers are back up (VM's still down). So I stopped all the dockers with: docker kill $(docker ps -q) 
This command started showing all the container ID's and just stopped after 19 ID's. Hasn't returned to the command line in over an hour. I also ran:/etc/rc.d/rc.php-fpm restart
That was to try and restart the web interface. The shutting down failed (I assume because it was already down) but that is also stuck at starting php-fpm for over an hour.

 

I don't know where the restore is at or what I should do. I don't want to reboot if it's still working on rebuild. It's been rebuilding since yesterday at 3 PM

 

Any help is appreciated.

 

 

[JorgeB]

See if you can get the syslog:

cp /var/log/syslog /boot/syslog.txt

 

[jay2068]

36 minutes ago, JorgeB said:

See if you can get the syslog:

cp /var/log/syslog /boot/syslog.txt

 

Attached is the log

unraid_syslog.txt

[jay2068]

I see btfrs error and it also says it requires a reboot

 

Mar  9 22:31:11 Tower12 kernel: BTRFS critical (device loop2): corrupt leaf: block=30805327872 slot=227 extent bytenr=30800609280 len=16384 invalid generation, have 9837406843266202086 expect (0, 1967407]
Mar  9 22:31:11 Tower12 kernel: BTRFS info (device loop2): leaf 30805327872 gen 1967406 total ptrs 249 free space 221 owner 2
Mar  9 22:31:11 Tower12 kernel:         item 0 key (30796890112 169 0) itemoff 16250 itemsize 33
Mar  9 22:31:11 Tower12 kernel:                 extent refs 1 gen 1843974 flags 2
Mar  9 22:31:11 Tower12 kernel:                 ref#0: tree block backref root 13899
Mar  9 22:31:11 Tower12 kernel:         item 1 key (30796906496 169 0) itemoff 16190 itemsize 60

 

and

 

Mar  9 22:31:11 Tower12 kernel:                 ref#0: tree block backref root 7
Mar  9 22:31:11 Tower12 kernel: BTRFS error (device loop2): block=30805327872 write time tree block corruption detected
Mar  9 22:31:13 Tower12 kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2377: errno=-5 IO failure (Error while writing out transaction)
Mar  9 22:31:13 Tower12 kernel: BTRFS info (device loop2): forced readonly
Mar  9 22:31:13 Tower12 kernel: BTRFS warning (device loop2): Skipping commit of aborted transaction.
Mar  9 22:31:13 Tower12 kernel: BTRFS: error (device loop2) in cleanup_transaction:1942: errno=-5 IO failure
Mar  9 22:31:13 Tower12 kernel: BTRFS warning (device loop2): Skipping commit of aborted transaction.
Mar  9 22:31:13 Tower12 kernel: BTRFS: error (device loop2) in cleanup_transaction:1942: errno=-5 IO failure

Mar 10 00:20:49 Tower12 kernel: Fixing recursive fault but reboot is needed!

 

I think I should reboot and see where I am at.

 

 

Edited March 10, 2023 by jay2068

[JorgeB]

Mar  9 22:42:41 Tower12 kernel: md: sync done. time=36575sec
Mar  9 22:42:41 Tower12 kernel: md: recovery thread: exit status: 0

Rebuild finished.

 

btrfs errors are about the docker image, delete and re-create, and try switching to ipvlan (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)), this might have been the source of the main issues.

[jay2068]

5 minutes ago, JorgeB said:

Mar  9 22:42:41 Tower12 kernel: md: sync done. time=36575sec
Mar  9 22:42:41 Tower12 kernel: md: recovery thread: exit status: 0

Rebuild finished.

 

btrfs errors are about the docker image, delete and re-create, and try switching to ipvlan (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)), this might have been the source of the main issues.

Ok I'll try a reboot and report back.

[jay2068]

Everyhting is back working. I modified the docker network type also.

Edited March 10, 2023 by jay2068