January 1, 201610 yr It started the monthly parity check this morning, and when I got up I noticed I could not connect to 2 of my user shares, but could still connect to all the others, could use my SABnzbd and Deluge dockers and my Windows VM was still working. A few hours later, I lost connectivity to a couple more user shares. Now, after about 11.5 hours since parity check started, I lost connectivity to my Windows VM, all my dockers and all but 2 of my user shares. The check is about 2/3 finished, and says it will complete in about 5 more hours, which is about normal. This has happened the last 3 months, and I ended up cancelling the parity check the last 2 months, because it's so annoying. I will probably let it finish today, and just forego my server until it finishes, but this didn't use to happen, so I'm hoping to find a solution which allows me to use my server all the time. The full diagnistics file is too large to post, so I opened it to see what was taking so much space. The docker log is huge, and very confusing. I don't have plex installed. I did several months ago, but didn't like it, so I (thought I) removed it completely from my system. But, here are the oldest, then most recent lines from the Docker log file... time="2015-12-24T12:47:12.356120960-05:00" level=error msg="Error from V2 registry: unable to copy v2 image blob data: unexpected EOF" time="2015-12-24T14:40:48.372545782-05:00" level=error msg="Handler for GET /containers/{name:.*}/json returned error: no such id: PlexMediaServer" time="2015-12-24T14:40:48.372574936-05:00" level=error msg="HTTP Error" err="no such id: PlexMediaServer" statusCode=404 time="2015-12-24T14:40:48.372885518-05:00" level=error msg="Handler for GET /images/{name:.*}/json returned error: No such image: PlexMediaServer" time="2015-12-24T14:40:48.372902575-05:00" level=error msg="HTTP Error" err="No such image: PlexMediaServer" statusCode=404 time="2015-12-25T21:31:50.248284353-05:00" level=error msg="Handler for GET /containers/{name:.*}/json returned error: no such id: PlexMediaServer" time="2015-12-25T21:31:50.248311595-05:00" level=error msg="HTTP Error" err="no such id: PlexMediaServer" statusCode=404 time="2015-12-25T21:31:50.248647337-05:00" level=error msg="Handler for GET /images/{name:.*}/json returned error: No such image: PlexMediaServer" time="2015-12-25T21:31:50.248666345-05:00" level=error msg="HTTP Error" err="No such image: PlexMediaServer" statusCode=404 time="2015-12-26T16:09:12.292030005-05:00" level=error msg="Handler for GET /containers/{name:.*}/json returned error: no such id: PlexMediaServer" time="2015-12-26T16:09:12.292057787-05:00" level=error msg="HTTP Error" err="no such id: PlexMediaServer" statusCode=404 time="2015-12-26T16:09:12.292426661-05:00" level=error msg="Handler for GET /images/{name:.*}/json returned error: No such image: PlexMediaServer" time="2015-12-26T16:09:12.292446754-05:00" level=error msg="HTTP Error" err="No such image: PlexMediaServer" statusCode=404 time="2015-12-27T22:40:15.194642218-05:00" level=error msg="Error from V2 registry: Get https://auth.docker.io/token?scope=repository%3Alinuxserver%2Fsickrage%3Apull&service=registry.docker.io: read tcp 54.165.33.192:443: i/o timeout" time="2015-12-30T12:41:29.612549950-05:00" level=error msg="Handler for GET /containers/{name:.*}/json returned error: no such id: PlexMediaServer" time="2015-12-30T12:41:29.612579436-05:00" level=error msg="HTTP Error" err="no such id: PlexMediaServer" statusCode=404 time="2015-12-30T12:41:29.613371920-05:00" level=error msg="Handler for GET /images/{name:.*}/json returned error: No such image: PlexMediaServer" time="2015-12-30T12:41:29.613391503-05:00" level=error msg="HTTP Error" err="No such image: PlexMediaServer" statusCode=404 time="2015-12-30T12:42:13.242116954-05:00" level=error msg="Handler for GET /containers/{name:.*}/json returned error: no such id: PlexMediaServer" time="2015-12-30T12:42:13.242143206-05:00" level=error msg="HTTP Error" err="no such id: PlexMediaServer" statusCode=404 time="2015-12-30T12:42:13.242489824-05:00" level=error msg="Handler for GET /images/{name:.*}/json returned error: No such image: PlexMediaServer" time="2015-12-30T12:42:13.242505957-05:00" level=error msg="HTTP Error" err="No such image: PlexMediaServer" statusCode=404 time="2015-12-31T15:49:50.383099639-05:00" level=error msg="Handler for GET /containers/{name:.*}/json returned error: no such id: PlexMediaServer" time="2015-12-31T15:49:50.383126710-05:00" level=error msg="HTTP Error" err="no such id: PlexMediaServer" statusCode=404 time="2015-12-31T15:49:50.383439731-05:00" level=error msg="Handler for GET /images/{name:.*}/json returned error: No such image: PlexMediaServer" time="2015-12-31T15:49:50.383456246-05:00" level=error msg="HTTP Error" err="No such image: PlexMediaServer" statusCode=404 time="2016-01-01T11:18:06.501777280-05:00" level=error msg="Handler for GET /containers/{name:.*}/json returned error: no such id: PlexMediaServer" time="2016-01-01T11:18:06.501804234-05:00" level=error msg="HTTP Error" err="no such id: PlexMediaServer" statusCode=404 time="2016-01-01T11:18:06.507105745-05:00" level=error msg="Handler for GET /images/{name:.*}/json returned error: No such image: PlexMediaServer" time="2016-01-01T11:18:06.507125312-05:00" level=error msg="HTTP Error" err="No such image: PlexMediaServer" statusCode=404 time="2016-01-01T12:18:48.598722719-05:00" level=error msg="Handler for GET /containers/{name:.*}/json returned error: no such id: PlexMediaServer" time="2016-01-01T12:18:48.598749567-05:00" level=error msg="HTTP Error" err="no such id: PlexMediaServer" statusCode=404 time="2016-01-01T12:18:48.599041078-05:00" level=error msg="Handler for GET /images/{name:.*}/json returned error: No such image: PlexMediaServer" time="2016-01-01T12:18:48.599058997-05:00" level=error msg="HTTP Error" err="No such image: PlexMediaServer" statusCode=404 time="2016-01-01T16:09:39.857660946-05:00" level=error msg="Failed to log msg \"Traceback (most recent call last):\" for logger json-file: write /var/lib/docker/containers/fca6b477bdabe809d472aa034f70122e0439df28dc27ed840affd2b516cff321/fca6b477bdabe809d472aa034f70122e0439df28dc27ed840affd2b516cff321-json.log: read-only file system" time="2016-01-01T16:09:39.857888593-05:00" level=error msg="Failed to log msg \" File \\\"/usr/lib/python2.7/site-packages/supervisor/loggers.py\\\", line 84, in emit\" for logger json-file: write /var/lib/docker/containers/fca6b477bdabe809d472aa034f70122e0439df28dc27ed840affd2b516cff321/fca6b477bdabe809d472aa034f70122e0439df28dc27ed840affd2b516cff321-json.log: read-only file system" time="2016-01-01T16:09:39.857998112-05:00" level=error msg="Failed to log msg \" File \\\"/usr/lib/python2.7/site-packages/supervisor/loggers.py\\\", line 64, in flush\" for logger json-file: write /var/lib/docker/containers/fca6b477bdabe809d472aa034f70122e0439df28dc27ed840affd2b516cff321/fca6b477bdabe809d472aa034f70122e0439df28dc27ed840affd2b516cff321-json.log: read-only file system" time="2016-01-01T16:09:39.858042615-05:00" level=error msg="Failed to log msg \"IOError: [Errno 30] Read-only file system\" for logger json-file: write /var/lib/docker/containers/fca6b477bdabe809d472aa034f70122e0439df28dc27ed840affd2b516cff321/fca6b477bdabe809d472aa034f70122e0439df28dc27ed840affd2b516cff321-json.log: read-only file system" time="2016-01-01T16:09:39.858061621-05:00" level=error msg="Failed to log msg \"Traceback (most recent call last):\" for logger json-file: write /var/lib/docker/containers/fca6b477bdabe809d472aa034f70122e0439df28dc27ed840affd2b516cff321/fca6b477bdabe809d472aa034f70122e0439df28dc27ed840affd2b516cff321-json.log: read-only file system" time="2016-01-01T16:09:39.858095797-05:00" level=error msg="Failed to log msg \" File \\\"/usr/lib/python2.7/site-packages/supervisor/loggers.py\\\", line 84, in emit\" for logger json-file: write /var/lib/docker/containers/fca6b477bdabe809d472aa034f70122e0439df28dc27ed840affd2b516cff321/fca6b477bdabe809d472aa034f70122e0439df28dc27ed840affd2b516cff321-json.log: read-only file system" ... time="2016-01-01T16:14:07.786416908-05:00" level=error msg="Failed to log msg \" File \\\"/usr/lib/python2.7/site-packages/supervisor/loggers.py\\\", line 64, in flush\" for logger json-file: write /var/lib/docker/containers/fca6b477bdabe809d472aa034f70122e0439df28dc27ed840affd2b516cff321/fca6b477bdabe809d472aa034f70122e0439df28dc27ed840affd2b516cff321-json.log: read-only file system" time="2016-01-01T16:14:07.793252541-05:00" level=error msg="Failed to log msg \"IOError: [Errno 30] Read-only file system\" for logger json-file: write /var/lib/docker/containers/fca6b477bdabe809d472aa034f70122e0439df28dc27ed840affd2b516cff321/fca6b477bdabe809d472aa034f70122e0439df28dc27ed840affd2b516cff321-json.log: read-only file system" time="2016-01-01T16:14:07.793298219-05:00" level=error msg="Failed to log msg \"Traceback (most recent call last):\" for logger json-file: write /var/lib/docker/containers/fca6b477bdabe809d472aa034f70122e0439df28dc27ed840affd2b516cff321/fca6b477bdabe809d472aa034f70122e0439df28dc27ed840affd2b516cff321-json.log: read-only file system" time="2016-01-01T16:14:07.793405418-05:00" level=error msg="Failed to log msg \" File \\\"/usr/lib/python2.7/site-packages/supervisor/loggers.py\\\", line 84, in emit\" for logger json-file: write /var/lib/docker/containers/fca6b477bdabe809d472aa034f70122e0439df28dc27ed840affd2b516cff321/fca6b477bdabe809d472aa034f here are the most recent lines from the syslog... Jan 1 16:14:07 media kernel: blk_update_request: 98035 callbacks suppressed Jan 1 16:14:07 media kernel: blk_update_request: I/O error, dev loop0, sector 1212192 Jan 1 16:14:07 media kernel: btrfs_dev_stat_print_on_error: 98036 callbacks suppressed Jan 1 16:14:07 media kernel: BTRFS: bdev /dev/loop0 errs: wr 9, rd 495459, flush 1, corrupt 0, gen 0 Jan 1 16:14:07 media kernel: blk_update_request: I/O error, dev loop0, sector 3309344 Jan 1 16:14:07 media kernel: BTRFS: bdev /dev/loop0 errs: wr 9, rd 495460, flush 1, corrupt 0, gen 0 Jan 1 16:14:07 media kernel: blk_update_request: I/O error, dev loop0, sector 1575840 Jan 1 16:14:07 media kernel: BTRFS: bdev /dev/loop0 errs: wr 9, rd 495461, flush 1, corrupt 0, gen 0 Jan 1 16:14:07 media kernel: blk_update_request: I/O error, dev loop0, sector 3672992 Jan 1 16:14:07 media kernel: BTRFS: bdev /dev/loop0 errs: wr 9, rd 495462, flush 1, corrupt 0, gen 0 Jan 1 16:14:07 media kernel: blk_update_request: I/O error, dev loop0, sector 1212192 Jan 1 16:14:07 media kernel: BTRFS: bdev /dev/loop0 errs: wr 9, rd 495463, flush 1, corrupt 0, gen 0 Jan 1 16:14:07 media kernel: blk_update_request: I/O error, dev loop0, sector 3309344 Jan 1 16:14:07 media kernel: BTRFS: bdev /dev/loop0 errs: wr 9, rd 495464, flush 1, corrupt 0, gen 0 Jan 1 16:14:07 media kernel: blk_update_request: I/O error, dev loop0, sector 1212192 Jan 1 16:14:07 media kernel: BTRFS: bdev /dev/loop0 errs: wr 9, rd 495465, flush 1, corrupt 0, gen 0 Jan 1 16:14:07 media kernel: blk_update_request: I/O error, dev loop0, sector 3309344 Jan 1 16:14:07 media kernel: BTRFS: bdev /dev/loop0 errs: wr 9, rd 495466, flush 1, corrupt 0, gen 0 Jan 1 16:14:07 media kernel: blk_update_request: I/O error, dev loop0, sector 1212192 Jan 1 16:14:07 media kernel: BTRFS: bdev /dev/loop0 errs: wr 9, rd 495467, flush 1, corrupt 0, gen 0 Jan 1 16:14:07 media kernel: blk_update_request: I/O error, dev loop0, sector 3309344 Jan 1 16:14:07 media kernel: BTRFS: bdev /dev/loop0 errs: wr 9, rd 495468, flush 1, corrupt 0, gen 0 Jan 1 16:14:12 media kernel: blk_update_request: 106949 callbacks suppressed Jan 1 16:14:12 media kernel: blk_update_request: I/O error, dev loop0, sector 202176 Jan 1 16:14:12 media kernel: btrfs_dev_stat_print_on_error: 106950 callbacks suppressed Jan 1 16:14:12 media kernel: BTRFS: bdev /dev/loop0 errs: wr 9, rd 602421, flush 1, corrupt 0, gen 0 Jan 1 16:14:12 media kernel: blk_update_request: I/O error, dev loop0, sector 2299328 Jan 1 16:14:12 media kernel: BTRFS: bdev /dev/loop0 errs: wr 9, rd 602422, flush 1, corrupt 0, gen 0 Jan 1 16:14:12 media kernel: blk_update_request: I/O error, something is definitely broke, but I'm not sure where to start
January 1, 201610 yr Without the full diagnostics (upload it to Dropbox or something) it's hard to say but it looks like your cache drive is dropping offline due to a hardware failure
January 1, 201610 yr Author I started looking on my Flash drive for any signs of Plex or anything else that I might not need, and I'm not sure if I need these folders under /plugins... dockerMan (I used to use this plugin, but it was replaced a while back with build in functionality, but I'm not sure if this folder is from the old plugin, or the new system) OpenELEC (I installed this a while back, but then removed it, therefore I just deleted this folder) virtMan (same as dockerMan, I know it was needed in the past, not sure it's needed now) I didn't see anything else that looked 'out of place' in my flash drive review. I just putty'd in to look at my cache drive, and both cache and user are shown in red with a question mark in front of them, and tell me "cannot read directory contents" when I try to open them. ideas?
January 1, 201610 yr Author While looking around some more, I saw this on the unassigned devices tab... Warning: file_put_contents(): Only 0 of 209 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 66 My flash drive is only 4% used, so it can't be out of space. is this related, or a separate issue I need to deal with?
January 2, 201610 yr While looking around some more, I saw this on the unassigned devices tab... Warning: file_put_contents(): Only 0 of 209 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 66 My flash drive is only 4% used, so it can't be out of space. is this related, or a separate issue I need to deal with? Its the tmpfs. Your log is basically a single line over and over again with an error from the smart monitoring process, which I believe is related to a drive having issues. But, on a quick glance, the drive with the problems appears to be your Kingston SSD (VM Drive attached through unassigned devices). Currently have company over, so can't fully analyse everything however.
February 1, 201610 yr Author This happened again this morning. About 6 hours after starting the parity check one of my user shares was unavailable. The unassigned drive is 'blank' so my dockers and my VMs are dead (they reside on the unassigned drive). I can't see any reason for parity check to even touch the unassigned drive. It is unprotected by design, so the parity check should not bother this drive in any way, as far as I can see. Anyway, I've stopped parity check now, as last time by the time it completed, I couldn't access the GUI, so I was forced to hard boot the server, which resulted in needing another parity check, so why bother. This has been going on for months, so I have no idea if my parity is even correct at this point. I need some help getting this issue fixed. media-diagnostics-20160201-0839.zip
Archived
This topic is now archived and is closed to further replies.