syrys Posted July 6, 2020 Share Posted July 6, 2020 Hey all, i think i need some serious help with my unraid (cache?). My unraid setup was running nice and smooth (minor unrelated hiccups) for few years now. I have about 10 drives, 1 parity, 1 unassigned drives drive (i use this for downloads etc), and 2x ssds (cache). since couple of months ago, ive been getting some weirdness. my SABnzbd docker was the first to complain (atleast that i noticed) with errors like this on every download: Traceback (most recent call last): File "/usr/share/sabnzbdplus/cherrypy/_cprequest.py", line 663, in respond self.body.process() File "/usr/share/sabnzbdplus/cherrypy/_cpreqbody.py", line 989, in process super(RequestBody, self).process() File "/usr/share/sabnzbdplus/cherrypy/_cpreqbody.py", line 558, in process proc(self) File "/usr/share/sabnzbdplus/cherrypy/_cpreqbody.py", line 223, in process_multipart_form_data process_multipart(entity) File "/usr/share/sabnzbdplus/cherrypy/_cpreqbody.py", line 215, in process_multipart part.process() File "/usr/share/sabnzbdplus/cherrypy/_cpreqbody.py", line 556, in process self.default_proc() File "/usr/share/sabnzbdplus/cherrypy/_cpreqbody.py", line 715, in default_proc self.file = self.read_into_file() File "/usr/share/sabnzbdplus/cherrypy/_cpreqbody.py", line 729, in read_into_file fp_out = self.make_file() File "/usr/share/sabnzbdplus/cherrypy/_cpreqbody.py", line 512, in make_file return tempfile.TemporaryFile() File "/usr/lib/python2.7/tempfile.py", line 511, in TemporaryFile (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags) File "/usr/lib/python2.7/tempfile.py", line 244, in _mkstemp_inner fd = _os.open(file, flags, 0600) OSError: [Errno 30] Read-only file system: '/tmp/tmpoyMrxR' This issue happens after a day or so of the system running, basically the unraid system somehow "breaks", and then no more downloads work (above error). Simply restarting the unraid server fixes it (restarting docker does not, sometimes it wont even allow restarting docker, says something went wrong). Anyway, last line of above error is interesting. Something is in read only mode? my cache drive maybe? Then exploring my unraid logs, i see things like this: BTRFS: error (device loop2) in btrfs_finish_ordered_io:3107: errno=-5 IO failure some btrfs issue? only btrfs usage is the 2x cache ssds i believe. so something going on with that? Ive googled the above error, and got to a place where people suggested that the cache might be corrupted. But i wasent really clear on how to know for sure, and how to go about fixing this. any help will be appreciated. i am a total noob with all this just to clarify. when i restart, everything works fine. ive disabled couple of dockers i use (radarr, deluge) which i dont use as much, and with those disabled the weirdness/errors are less frequent. so basically, when i restart the server, everything works fine (other dockers run fine: plex, sonarr, sabnzbd), but after couple of days of running, i notice weirdness like sonarr no longer imports files, plex sometimes crashes and dies, sab sometimes fails to download anything (first error i mentioned). Quote Link to comment
Kevek79 Posted July 6, 2020 Share Posted July 6, 2020 I might not be a lot of help myself, but I am pretty sure that those who could help you will need your diagnostics file ;). Please attach to your next posting. 1 Quote Link to comment
trurl Posted July 6, 2020 Share Posted July 6, 2020 13 minutes ago, syrys said: BTRFS: error (device loop2) in btrfs_finish_ordered_io:3107: errno=-5 IO failure some btrfs issue? only btrfs usage is the 2x cache ssds i believe. so something going on with that? That particular btrfs device is your docker img vdisk. It probably lives on your cache though so there might also be a problem there. Go to Tools - Diagnostics and attach the complete Diagnostics ZIP file to your NEXT post. Quote Link to comment
syrys Posted July 17, 2020 Author Share Posted July 17, 2020 (edited) On 7/7/2020 at 1:34 AM, Kevek79 said: I might not be a lot of help myself, but I am pretty sure that those who could help you will need your diagnostics file ;). Please attach to your next posting. On 7/7/2020 at 1:46 AM, trurl said: That particular btrfs device is your docker img vdisk. It probably lives on your cache though so there might also be a problem there. Go to Tools - Diagnostics and attach the complete Diagnostics ZIP file to your NEXT post. Sorry for the late response. I waited a little bit so the issue triggers again so i can get the exact error and get the diagnostic file when it happens. week ago when the server was having similar issues, it didnt even let me download the diagnostic file, sigh. Anyway, same error happened again. Here are some errors on the system log: Jul 15 01:08:35 karie kernel: print_req_error: I/O error, dev sdj, sector 6078419056 Jul 17 01:05:49 karie kernel: BTRFS: error (device loop2) in btrfs_finish_ordered_io:3107: errno=-5 IO failure hmm, unsure what the first error is either. sdj is my unassigned device drive, the drive i have all my downloads going to (and then they get picked up and moved to the array, nothing important in sdj). Any help is appreciated. Thanks in advance karie-diagnostics-20200717-1249.zip Edited July 17, 2020 by syrys Quote Link to comment
JorgeB Posted July 17, 2020 Share Posted July 17, 2020 Docker image is corrupt, delete and re-create, also due to a previous bug your cache pool is not redundant, see here. Quote Link to comment
syrys Posted July 17, 2020 Author Share Posted July 17, 2020 (edited) 16 hours ago, johnnie.black said: Docker image is corrupt, delete and re-create, also due to a previous bug your cache pool is not redundant, see here. oh wow, that sounds worrying. alright, ill follow cache pool issue instructions from that link. as for the docker image recreation, are there any instructions i should be following for this? or is it basically manually re creating a new docker, and re downloading all the previous dockers ive been using (plex, sonarr, etc) and setting up the same docker settings for each of them as before? Edited July 17, 2020 by syrys Quote Link to comment
JorgeB Posted July 18, 2020 Share Posted July 18, 2020 8 hours ago, syrys said: as for the docker image recreation, are there any instructions i should be following for this? See: https://forums.unraid.net/topic/57181-docker-faq/?do=findComment&comment=564309 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.