Help: BTRFS error (device loop2) IO failure


Recommended Posts

Hey all,

 

i think i need some serious help with my unraid (cache?). My unraid setup was running nice and smooth (minor unrelated hiccups) for few years now. I have about 10 drives, 1 parity, 1 unassigned drives drive (i use this for downloads etc), and 2x ssds (cache).

 

since couple of months ago, ive been getting some weirdness. my SABnzbd docker was the first to complain (atleast that i noticed) with errors like this on every download:
 

Traceback (most recent call last):
  File "/usr/share/sabnzbdplus/cherrypy/_cprequest.py", line 663, in respond
    self.body.process()
  File "/usr/share/sabnzbdplus/cherrypy/_cpreqbody.py", line 989, in process
    super(RequestBody, self).process()
  File "/usr/share/sabnzbdplus/cherrypy/_cpreqbody.py", line 558, in process
    proc(self)
  File "/usr/share/sabnzbdplus/cherrypy/_cpreqbody.py", line 223, in process_multipart_form_data
    process_multipart(entity)
  File "/usr/share/sabnzbdplus/cherrypy/_cpreqbody.py", line 215, in process_multipart
    part.process()
  File "/usr/share/sabnzbdplus/cherrypy/_cpreqbody.py", line 556, in process
    self.default_proc()
  File "/usr/share/sabnzbdplus/cherrypy/_cpreqbody.py", line 715, in default_proc
    self.file = self.read_into_file()
  File "/usr/share/sabnzbdplus/cherrypy/_cpreqbody.py", line 729, in read_into_file
    fp_out = self.make_file()
  File "/usr/share/sabnzbdplus/cherrypy/_cpreqbody.py", line 512, in make_file
    return tempfile.TemporaryFile()
  File "/usr/lib/python2.7/tempfile.py", line 511, in TemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
  File "/usr/lib/python2.7/tempfile.py", line 244, in _mkstemp_inner
    fd = _os.open(file, flags, 0600)
OSError: [Errno 30] Read-only file system: '/tmp/tmpoyMrxR'

This issue happens after a day or so of the system running, basically the unraid system somehow "breaks", and then no more downloads work (above error). Simply restarting the unraid server fixes it (restarting docker does not, sometimes it wont even allow restarting docker, says something went wrong). Anyway, last line of above error is interesting. Something is in read only mode? my cache drive maybe?

Then exploring my unraid logs, i see things like this:

BTRFS: error (device loop2) in btrfs_finish_ordered_io:3107: errno=-5 IO failure

some btrfs issue? only btrfs usage is the 2x cache ssds i believe. so something going on with that?

Ive googled the above error, and got to a place where people suggested that the cache might be corrupted. But i wasent really clear on how to know for sure, and how to go about fixing this. any help will be appreciated. i am a total noob with all this :( 

 

just to clarify. when i restart, everything works fine. ive disabled couple of dockers i use (radarr, deluge) which i dont use as much, and with those disabled the weirdness/errors are less frequent. so basically, when i restart the server, everything works fine (other dockers run fine: plex, sonarr, sabnzbd), but after couple of days of running, i notice weirdness like sonarr no longer imports files, plex sometimes crashes and dies, sab sometimes fails to download anything (first error i mentioned).

Link to comment
13 minutes ago, syrys said:

BTRFS: error (device loop2) in btrfs_finish_ordered_io:3107: errno=-5 IO failure

some btrfs issue? only btrfs usage is the 2x cache ssds i believe. so something going on with that?

That particular btrfs device is your docker img vdisk. It probably lives on your cache though so there might also be a problem there.

 

Go to Tools - Diagnostics and attach the complete Diagnostics ZIP file to your NEXT post.

Link to comment
  • 2 weeks later...
On 7/7/2020 at 1:34 AM, Kevek79 said:

I might not be a lot of help myself, but I am pretty sure that those who could help you will need your diagnostics file ;).

Please attach to your next posting.

 

On 7/7/2020 at 1:46 AM, trurl said:

That particular btrfs device is your docker img vdisk. It probably lives on your cache though so there might also be a problem there.

 

Go to Tools - Diagnostics and attach the complete Diagnostics ZIP file to your NEXT post.

 

Sorry for the late response. I waited a little bit so the issue triggers again so i can get the exact error and get the diagnostic file when it happens. week ago when the server was having similar issues, it didnt even let me download the diagnostic file, sigh. 


Anyway, same error happened again. Here are some errors on the system log:

Jul 15 01:08:35 karie kernel: print_req_error: I/O error, dev sdj, sector 6078419056 
Jul 17 01:05:49 karie kernel: BTRFS: error (device loop2) in btrfs_finish_ordered_io:3107: errno=-5 IO failure

hmm, unsure what the first error is either. sdj is my unassigned device drive, the drive i have all my downloads going to (and then they get picked up and moved to the array, nothing important in sdj). 

 

Any help is appreciated.

Thanks in advance :)
 

karie-diagnostics-20200717-1249.zip

Edited by syrys
Link to comment
16 hours ago, johnnie.black said:

Docker image is corrupt, delete and re-create, also due to a previous bug your cache pool is not redundant, see here.

oh wow, that sounds worrying. alright, ill follow cache pool issue instructions from that link. 

as for the docker image recreation, are there any instructions i should be following for this? or is it basically manually re creating a new docker, and re downloading all the previous dockers ive been using (plex, sonarr, etc) and setting up the same docker settings for each of them as before?

Edited by syrys
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.