Please Help! Multiple Problems out of the blue

[JasenHicks]

1. Went to watch some TV from my PLEX instance on my UNRAD today and it wouldn't connect.

2. Remoted into the WebUI for unraid and saw PLEX wasn't running.

3. Tried to start it... gave me the Error 403 screen (not helpful for me of course).

4. Looked at the log had some json error thing. Did a google search and found a lot of old threads that weren't similar enough to my problem.

5. Rebooted the system, it all came back up and then Plex started. Tried the WebUI for plex. Failed to load.

6. Tried to remove the Plex Docker container and start a new one. Wouldn't delete.

7. Just got this: "Drive mounted read-only or completely full. Begin Investigation Here:" on my cache drive (500GB SSD x2 @  76%, BRFTS)

8. Ran mover. No Change.

9. Ran Fix Common Problems extended test. Showed this as well: "Unable to write to Docker Image", further  it said: Docker Image either full or corrupted. Investigate Here.

10. Docker image is 21.5GB.

11. Also noticed: "Share appdata set to cache-only, but files / folders exist on the array" this has been a thing since I set up my unraid. Never did anything with it.

 

Now, I can't remove the plex docker. I can't start it either. If I try to look at the logs it says I cant look at them because they were marked for removal.

 

Diagnostics attached. I have no idea what to do at this point, nor do I know why my cache drive is so full.

zeus-diagnostics-20190806-1115 (1).zip

[JasenHicks]

And when I reboot... I notice this happens:

 

"Docker Service failed to start."

[JorgeB]

Docker image is corrupt, delete and re-create.

[JasenHicks]

So, you say this but what are the ramifications? Is there a tutorial? Am I going to have to reset up everything like SAB, SONAR, RADAR, PLEX, etc?

[JorgeB]

https://forums.unraid.net/topic/57181-real-docker-faq/?do=findComment&comment=564309

 

[JasenHicks]

Thanks. That worked to get my Dockers back and things running again. Now I am getting random errors writing to the shares specifically that the media is WRITE PROTECTED. I assume this has something to do with the CACHE DRIVE at 76%?

 

 

[JasenHicks]

ErrorWarningSystemArrayLogin


Aug 7 03:02:54 Zeus kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
Aug 7 03:02:54 Zeus kernel: loop: Write error at byte offset 8208318464, length 4096.
Aug 7 03:02:54 Zeus kernel: print_req_error: I/O error, dev loop3, sector 16031872
Aug 7 03:02:54 Zeus kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
Aug 7 03:02:54 Zeus kernel: loop: Write error at byte offset 8569929728, length 4096.
Aug 7 03:02:54 Zeus kernel: print_req_error: I/O error, dev loop3, sector 16738144
Aug 7 03:02:54 Zeus kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
Aug 7 03:02:54 Zeus kernel: loop: Write error at byte offset 8571240448, length 4096.
Aug 7 03:02:54 Zeus kernel: print_req_error: I/O error, dev loop3, sector 16740704
Aug 7 03:02:54 Zeus kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
Aug 7 03:02:54 Zeus kernel: loop: Write error at byte offset 3206844416, length 4096.
Aug 7 03:02:54 Zeus kernel: print_req_error: I/O error, dev loop3, sector 6263368
Aug 7 03:02:54 Zeus kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
Aug 7 03:02:54 Zeus kernel: loop: Write error at byte offset 8569667584, length 4096.
Aug 7 03:02:54 Zeus kernel: print_req_error: I/O error, dev loop3, sector 16737632
Aug 7 03:02:54 Zeus kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
Aug 7 03:02:54 Zeus kernel: loop: Write error at byte offset 3206844416, length 4096.
Aug 7 03:02:54 Zeus kernel: print_req_error: I/O error, dev loop3, sector 6263368
Aug 7 03:02:54 Zeus kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
Aug 7 03:02:54 Zeus kernel: loop: Write error at byte offset 8569667584, length 4096.
Aug 7 03:02:54 Zeus kernel: print_req_error: I/O error, dev loop3, sector 16737632
Aug 7 03:02:54 Zeus kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
Aug 7 03:02:54 Zeus kernel: BTRFS: error (device loop3) in btrfs_commit_transaction:2236: errno=-5 IO failure (Error while writing out transaction)
Aug 7 03:02:54 Zeus kernel: BTRFS info (device loop3): forced readonly
Aug 7 03:02:54 Zeus kernel: BTRFS warning (device loop3): Skipping commit of aborted transaction.
Aug 7 03:02:54 Zeus kernel: BTRFS: error (device loop3) in cleanup_transaction:1846: errno=-5 IO failure
Aug 7 03:02:54 Zeus kernel: BTRFS info (device loop3): delayed_refs has NO entry
Aug 7 03:04:38 Zeus kernel: btrfs_print_data_csum_error: 169 callbacks suppressed
Aug 7 03:04:38 Zeus kernel: BTRFS warning (device sdc1): csum failed root 5 ino 6687498 off 32803729408 csum 0x6670962f expected csum 0xe6d50113 mirror 1
Aug 7 03:04:38 Zeus kernel: BTRFS warning (device sdc1): csum failed root 5 ino 6687498 off 32803729408 csum 0x6670962f expected csum 0xe6d50113 mirror 1
Aug 7 03:04:38 Zeus kernel: BTRFS warning (device sdc1): csum failed root 5 ino 6687498 off 32803729408 csum 0x6670962f expected csum 0xe6d50113 mirror 2
Aug 7 03:04:39 Zeus kernel: BTRFS warning (device sdc1): csum failed root 5 ino 6687498 off 32803729408 csum 0x6670962f expected csum 0xe6d50113 mirror 1
Aug 7 03:04:39 Zeus kernel: BTRFS warning (device sdc1): csum failed root 5 ino 6687498 off 32803729408 csum 0x6670962f expected csum 0xe6d50113 mirror 2
Aug 7 03:04:39 Zeus kernel: BTRFS warning (device sdc1): csum failed root 5 ino 6687498 off 32803729408 csum 0x6670962f expected csum 0xe6d50113 mirror 1
Aug 7 03:04:39 Zeus kernel: BTRFS warning (device sdc1): csum failed root 5 ino 6687498 off 32803729408 csum 0x6670962f expected csum 0xe6d50113 mirror 2
Aug 7 03:04:39 Zeus kernel: BTRFS warning (device sdc1): csum failed root 5 ino 6687498 off 32803729408 csum 0x6670962f expected csum 0xe6d50113 mirror 1
Aug 7 03:04:39 Zeus kernel: BTRFS warning (device sdc1): csum failed root 5 ino 6687498 off 32803729408 csum 0x6670962f expected csum 0xe6d50113 mirror 2
Aug 7 03:04:40 Zeus move: error: move, 397: No such file or directory (2): lstat: /mnt/cache/downloads/incomplete/Terrace.House.Tokyo.2019.2020.S01E11.Broccoli.Pasta.Carbonara.Style.1080p.NF.WEB-DL.DD.2.0.x264-AJP69-RakuvArrow/__ADMIN__/SABnzbd_nzf_P7to1m

 

[JorgeB]

Please post the diagnostics: Tools -> Diagnostics

[JasenHicks]

Latest pull. 

zeus-diagnostics-20190807-1042.zip

[JasenHicks]

Interesting... I just noticed this. I'm certain the LOG shouldn't be 100%

[JorgeB]

Cache filesystem is also corrupt, best way forward is to backup all cache data, format and restore data.

 

Since there are checksums errors you'll want to make sure RAM is good, run memtest and keep in mind that overclocked RAM with Ryzen/Threadripper is in some cases known to corrupt data.

[JasenHicks]

Whelp... I think I'm screwed. I tried to run BACKUP from SETTINGS > BACKUP/RESTORE APP DATA. Ran through the process and checked the back up folder only to find it was empty. Any thoughts on where to go from here? I have a 500GB nVME drive sitting there ready to use as the cache drive but I really don't want to lose the data on the cache if I dont have to (I really dont want to setup all the plex, etc. stuff again).

[JorgeB]

Sorry, no experience with the appdata backup/restore plugin.

[JasenHicks]

Is there an alternative way you recommend backing up the cache drive? I am a bit out of my element on this one since my server was running fine for so long and now its just causing so many headaches and problems.

[JorgeB]

I take a daily snapshot then do a send/receive to a different device, most is explained here:

https://forums.unraid.net/topic/51703-vm-faq/?do=findComment&comment=523800