• Downgraded back to 6.6.7 due to Sqlite corruption


    limetech
    • Solved Urgent

    This is continuation of this original issue report.

     

    There is some conflicting information in that topic and, therefore, with your help, we want to try and get to the bottom of this.  We have several test servers running trying to reproduce this problem, so far hasn't happened.

     

    The first thing we want to definitively establish is whether the problem is isolated to only those cases where /config inside the container is mapped to:

    /mnt/user/appdata

    vs.

    /mnt/cache/appdata

    or

    /mnt/diskN/appdata

     

    Note: it does not matter what the "Default appdata storage location" is set to on Settings/Docker page.  Of interest is the actual path used by an individual container.

     

    What I'd like to ask is this:  For those of you who have seen this DB corruption occur after updating from Unraid 6.6.x to 6.7.x, please note the placement of your 'appdata' folder and how it's mapped to your problematic container.  Then update to 6.7.2 and run normally until you see corruption.  At that point, please capture diagnostics and report how appdata is being mapped.  If this is a huge PITA for you, then bow out of this testing - we are not trying to cause big disruptions for you.

     

    Finally I want to make clear: we are not "blaming" this issue on any other component, h/w or s/w, and also not "admitting" this is our problem.  We have not changed any code we're responsible for that would account for this, which doesn't mean we don't have a bug somewhere - probably we do, and a change in the kernel or some other component might simply be exposing some long hidden issue.  We are only interested in identifying and fixing the problem.

    • Upvote 4



    User Feedback

    Recommended Comments



    41 minutes ago, runraid said:

    Guys. Find another solution. It appears this company doesn’t care that your data is corrupting judging by their lack of responses. It’s time to move on. 

    I don't believe that there is any truth to that.

    • Upvote 1
    Link to comment
    5 hours ago, runraid said:

    It’s more than a user or two that suffers from corruption on /mnt/diskX. That didn’t work for me.

    While this may seem argumentative (and I don't mean it to be), but can you confirm this

     

    Your post on June 5th says the same thing

    On 6/5/2019 at 10:35 PM, runraid said:

    I just got corrupted again, even when routing to /mnt/disk3.

    but the only diagnostics that you've posted recently that I found was from June 9th (https://forums.unraid.net/topic/80680-checking-for-docker-app-updates-spins-forever/?tab=comments#comment-750259) which shows that appdata is stored on cache, disk1, 2, and 3.  While this is perfectly valid if you only deleted the plex folder (you did delete it didn't you?), it does *imply* that the possibility exists that the entire plex folder doesn't exist on disk3.

     

    And, according to the diagnostics on June 9th, your BIOS version is still from 2015, but numerous updates have been made to it, with the most recent being June 25, 2019.  Not saying that this is the issue, but an update would at least hopefully rule that out.

    Edited by Squid
    • Upvote 1
    Link to comment

    @Squid I'm basing that off of my experience along with seeing posts on here and reddit.

     

    I'm fully on /mnt/cache now. I haven't had a single problem since. 

     

    Yes, my bios is old, but as I mentioned before (forums are hard because they aren't threaded), I'm out of town for another few weeks and can't upgrade the bios until I get back.

    Link to comment

    So I moved my appdata share to Disk1.  I moved it all with the unbalance tool, and got it all moved to over /mnt/disk1/appdata.  I changed the share so that it only includes disk1.  Then I changed all of the dockers to point to it:  Plex, Sonarr, Sabnzbd.  

     

    I left for a little over an hour...and the Plex db corrupted again.  

     

    @Squid  What now???

     

    swissarmy-diagnostics-20190714-2311.zip

    Screen Shot 2019-07-14 at 6.16.43 PM.png

    Link to comment

    The share is located on disk 1 and disk 2.  

     

    Try this.  Make a new share for the hell of it.  Only include disk 1.  Point Plex to it (/mnt/disk1/NewShare) and see what happens.  (IE: Start Plex over at the new share.  Don't copy any of the old appdata to it)

    Edited by Squid
    Link to comment
    2 hours ago, runraid said:

    I'm fully on /mnt/cache now. I haven't had a single problem since. 

    I have a feeling though that if the share was fully contained onto a single drive you wouldn't have had any problems with diskX

     

    2 hours ago, runraid said:

    along with seeing posts on here and reddit

    Hearsay isn't usually helpful.

    Edited by Squid
    Link to comment
    18 minutes ago, Squid said:

    The share is located on disk 1 and disk 2.  

     

    Try this.  Make a new share for the hell of it.  Only include disk 1.  Point Plex to it (/mnt/disk1/NewShare) and see what happens.  (IE: Start Plex over at the new share.  Don't copy any of the old appdata to it)

     

    So if that is the case, then I don't know how to change it.  I had removed all disks from the share after I moved the data...and before I reset plex.

     

    I do what you asked....but somehow I think something else is going on.  

    Screen Shot 2019-07-14 at 8.01.53 PM.png

    Link to comment
    1 minute ago, Rich Minear said:

    I do what you asked....but somehow I think something else is going on.  

     

    Keep us informed...

    Link to comment

    @Squid 

     

    > Hearsay isn't usually helpful.

     

    Yup. I'm done with you guys. Deleting this account. I can feel your hostility towards us. 

    Link to comment
    15 hours ago, runraid said:

    @Squid 

     

    > Hearsay isn't usually helpful.

     

    Yup. I'm done with you guys. Deleting this account. I can feel your hostility towards us. 

     

    What hostility?  I haven't seen any from anyone but yourself.  We are hard at work trying to reproduce and resolve this issue, but you seem to think that because we haven't yet, that we're sitting here just twiddling our thumbs.  We are not.  We have multiple test servers constantly running Plex and injecting new data to it to try and force corruption.  Hasn't happened to us once.  That leads us to believe that this may be specific to individual setups/hardware, but we haven't figured out why just yet.

     

    You have a completely valid method to get back to a working state:  roll back to the 6.6.7 release.  Otherwise we are continuing to do testing and will provide more for folks to try in this bug report thread as we have ideas to narrow this down.

     

    Clearly this issue isn't as widespread as some may think, otherwise I think we'd have an outpouring of users and this thread would be a lot longer than 4 pages at this point.  That said, it is a VERY valid concern that we are very focused on resolving, but sometimes things take longer to fix.

    • Like 1
    • Upvote 2
    Link to comment
    2 hours ago, jonp said:

     

    What hostility?  I haven't seen any from anyone but yourself.  We are hard at work trying to reproduce and resolve this issue, but you seem to think that because we haven't yet, that we're sitting here just twiddling our thumbs.  We are not.  We have multiple test servers constantly running Plex and injecting new data to it to try and force corruption.  Hasn't happened to us once.  That leads us to believe that this may be specific to individual setups/hardware, but we haven't figured out why just yet.

     

    You have a completely valid method to get back to a working state:  roll back to the 6.6.7 release.  Otherwise we are continuing to do testing and will provide more for folks to try in this bug report thread as we have ideas to narrow this down.

     

    Clearly this issue isn't as widespread as some may think, otherwise I think we'd have an outpouring of users and this thread would be a lot longer than 4 pages at this point.  That said, it is a VERY valid concern that we are very focused on resolving, but sometimes things take longer to fix.

    @jonp  While I'm not ready to jump to hostility, there has absolutely been enough subtle (and not so subtle) language pointed as some of us to make us feel belittled or silly.  I was the one who originally started the original thread, about downgrading due to the corruption.  I was essentially told I was spreading information that wasn't true.  As a newbie, that is a little tough to take.  I was willing to try things to help with the testing...but it got to a point where the corruption was happening enough I had to stop.  Then the 6.7.2 came out, and this moved to bug fix.  Several of us were called out to help with the testing...which I did.  But then I heard nothing back for a while...and appears that no one was looking at the diagnostics I was uploading.  So I took a bit of a stronger tone in my messages...to get some attention.  

    I realize this is a very hard thing to troubleshoot.  But there ARE some people who are quick to point to our situation or configuration as the issue...even when multiple people are having the problem.  And they tell us to make changes, without explaining the changes or how to do it.  

    I do feel a bit like this has gone for a while with no official update or even a comment on what the findings have been so far...which leaves a number of us feeling a bit neglected.  

    That said...I have been making the changes that are asked...and still seeing corruption.  

     

     

    Link to comment
    Quote
    2 hours ago, jonp said:

     

    What hostility?  I haven't seen any from anyone but yourself.  We are hard at work trying to reproduce and resolve this issue, but you seem to think that because we haven't yet, that we're sitting here just twiddling our thumbs.  We are not.  We have multiple test servers constantly running Plex and injecting new data to it to try and force corruption.  Hasn't happened to us once.  That leads us to believe that this may be specific to individual setups/hardware, but we haven't figured out why just yet.

     

    You have a completely valid method to get back to a working state:  roll back to the 6.6.7 release.  Otherwise we are continuing to do testing and will provide more for folks to try in this bug report thread as we have ideas to narrow this down.

     

    Clearly this issue isn't as widespread as some may think, otherwise I think we'd have an outpouring of users and this thread would be a lot longer than 4 pages at this point.  That said, it is a VERY valid concern that we are very focused on resolving, but sometimes things take longer to fix.

     

    @jonp when users are posting log after log, and no one is replying with any actual update, not even "We still don't see the problems in these logs...", it gets a little frustrating. This is a product that folks here paid for, and blaming people's hardware/setup with 0 evidence is a little hard to swallow when rolling back a build fixes it. Adding a cache drive also fixes it.

     

    Do you all still need more logs or not?

    Link to comment

    Honestly for me guys, it's come and go on this DB crashing thing.  Plex has been working fairly stable from what I can tell (my watching, and other family members with no complaints, same house and remote).  But I do see this message in my Plex docker log from time to time:

     

    Sqlite3: Sleeping for 200ms to retry busy DB

    It's not there now, and someone is watching something so frankly, idk wtf is going on.

    Link to comment

    What's the best practice for deleting and rebuilding a corrupted Plex database? Mine has corrupted again after running fine for several weeks.... Hopefully this gets fixed soon, my wife is about to kill me over this.

    Link to comment
    18 hours ago, BBLV said:

    What's the best practice for deleting and rebuilding a corrupted Plex database? Mine has corrupted again after running fine for several weeks.... Hopefully this gets fixed soon, my wife is about to kill me over this.

    Take a look at the location of the database on the file system.  You can restore from one of the backups that should be there....and it would only be a day or two old.  Rebuilding it from scratch is a bummer....

    Link to comment
    22 hours ago, BBLV said:

    What's the best practice for deleting and rebuilding a corrupted Plex database? Mine has corrupted again after running fine for several weeks.... Hopefully this gets fixed soon, my wife is about to kill me over this.

    hey mate, plex does automatic task backups

     

    This should help https://support.plex.tv/articles/202485658-restore-a-database-backed-up-via-scheduled-tasks/

     

    In regards to this error, due to the lack of support I had to roll back, the forums are being spammed with change to diskx, from my reading most of us have tried this and failed.. has anyone actually had success with this workaround? (changing to diskx) 

     

    Seems adding a cache is the only working workaround, which is very annoying - because for me to do this I need to buy a expander card and SSD's

     

    I have done this but I don't think I'm going to upgrade when I have them installed until I see more movement on a software side of things

    Link to comment

    It happened to me! (not actually excited)  I'm adding a significant amount of media at the moment and I got a server down notification from Tautulli.

     

    Sqlite3: Sleeping for 200ms to retry busy DB.
    terminate called after throwing an instance of 'soci::soci_error'
    what(): sqlite3_statement_backend::loadOne: database is locked
    ****** PLEX MEDIA SERVER CRASHED, CRASH REPORT WRITTEN: /config/Library/Application Support/Plex Media Server/Crash Reports/1.16.2.1297-4b7ace214/PLEX MEDIA SERVER/4b038dd7-6f17-16f2-6b6beba4-37a6c664.dmp
    Sqlite3: Sleeping foStarting Plex Media Server.

    That's what my log on the Plex docker looks like right now

     

    I'm not sure what to do with the crash reports from Plex though...

     

    Attached is a very current diagnostics from unraid.

     

    Also I just restarted the docker and it looks like it is back up and running as normal.

    unraidserver-diagnostics-20190717-0212.zip

    Edited by ps2sunvalley
    Link to comment

    I just did a complete rebuilt CPU, MOBO and RAM (1155 to 1151-CL) and the corruption is still happening, so unless the corruption is hardware specific due to PSU and HDD i have ruled out the rest (i don't have anything else plugged in).

     

    I do have a SSD coming in just so i don't have to put up with the corruption (Got to keep the wife happy when Plex goes down and i don't have a PC near me to sort it out!), though my corruption with this not as frequent as some users (had my first yesterday which is close to 2 weeks since i did the rebuilt and only because i downloaded a whole new show and there was about 50 episodes that plex had to add in a short span of time).

     

    ps. I have also tried moving to DiskX and it still gets corrupted so lets hope Cache drive fix it - otherwise ill just have to permanently downgrade to 6.6.7 (would miss the nice UI :( )

    Edited by RaihaX
    Link to comment
    4 hours ago, Addy said:

    In regards to this error, due to the lack of support I had to roll back, the forums are being spammed with change to diskx, from my reading most of us have tried this and failed.. has anyone actually had success with this workaround? (changing to diskx) 

     

    Seems adding a cache is the only working workaround, which is very annoying - because for me to do this I need to buy a expander card and SSD's

    I'm unable to use a cache drive as well but I was able to switch over to using disk1 and so far I've not had any more corruption. I used UnBalance to move the appdata folder to disk1, updated the share to only include disk1, and finally rebuilt all the containers to use /mnt/disk1/appdata and so far things have been stable. It's only been a couple of days but fingers crossed.

     

    Link to comment

    HI all

    I experienced the same problems with Plex & Sonarr and SQLite DB corruptions when I first upgraded from 6.6.7 to 6.7.0

    Original setup for all dockers was /mnt/user/appdata, and moved to everything to /mnt/disk1/appdata where the issue was still appearing. I did not test an /mnt/cache as currently don't have a cache drive installed. I also changed the setup for all dockers from /mnt/user to /mnt/disk1.

    As per my experience

    /mnt/user/appdata - corruptions were appearing after 20-30 mins.

    /mnt/disk1/appdata - corruptions were appearing usually after 4-12 hours. Think once made it through a whole day but I am constantly adding my old digitized media to the libraries, so 

     

    Additional things I noticed

    i. Dockers running with paths linked to /mnt/user suffered from a severe performance degradation compared to /mnt/disk1. This could be noticed both on docker startup (the web ui is available much faster) and during normal operations. This to me is a very strange behaviour as /mnt/user/appdata was actually setup to be residing only in disk1, as that was my user share setup. I would expect some degradation but this was huge. Deluge web-ui kept loosing connection to the server, sonnar kept loosing connection to deluge and reported  that no download clients were available etc. Database rescans from both Sonarr and Plex were talking a very long time during which unRaid seemed to become very slow. After I moved to /mnt/disk1/appdata the performance issue disappeared. My libraries are huge and are still using /mnt/user paths so the performance issue was linked to the location of appdata directory where the DBs reside.

    ii. I noticed in Sonarr logs I was getting a lot of messages about the DB being locked. I did not check the Plex logs, but this seems to be similar to what people in this thread have mentioned with the Plex busy DB sleeping for 200ms message

    iii. It seems that the corruptions were mostly manifesting when there was severe load on SQLite or the dockers were running background processes that were also accessing/updating the DB. For example PLEX would remain stable. Once I added new content to the libraries and a library re-scan was initiated (which is running in the background) then after a while I got a corrupted DB. Similar for Sonarr after the initial DB refresh on startup, when the next refresh started (after 12 hours), again running in the background then the problem was appearing.

     

    I downgraded back to 6.6.7 after the 2nd week, as I am travelling most of the week and really don't have time to spent restoring/rebuilding the libraries or trying to resolve the issue.

     

    Unfortunately, I have no unRaid logs to share from that period.

     

     

     

    Link to comment

    FYI I'm running Binhex Plex Pass and still getting DB corruptions. I do NOT have Radarr or Sonarr like many others have who are also experiencing these issues.

     

    Seems to only happen while adding new media. I have gone days/weeks w/o corruption in between media adds... This is literally the only commonality that I can pinpoint between my DB corruptions.

    • Like 1
    Link to comment
    On 7/16/2019 at 9:36 PM, mdeabreu said:

    I'm unable to use a cache drive as well but I was able to switch over to using disk1 and so far I've not had any more corruption. I used UnBalance to move the appdata folder to disk1, updated the share to only include disk1, and finally rebuilt all the containers to use /mnt/disk1/appdata and so far things have been stable. It's only been a couple of days but fingers crossed.

     

    Looks like I cursed myself, got corruption for both Sonarr and Radarr despite using /mnt/disk1/appdata for the docker images. Attached are some diagnostics.

    Someone was streaming on Plex, there were some downloads, and Radarr was putting new media into place at which point there was the corruption.

     

    Edit: Unfortunately I’ve had to downgrade to 6.6.7, I was getting corruptions while attempting to rebuild the Radarr database and got frustrated. I hope we can see a software fix for this soon but in the mean time I’ll be sticking with 6.6.7 which seems to be stable for now. I’ll be watching this thread and happy to answer any questions about my setup if it’ll help with this issue.

    tower-diagnostics-20190722-0049.zip

    Edited by mdeabreu
    Link to comment

    I too have experienced the corruption issue with my Plex DBs.  Sooooo, I took the first suggestion and changed my appdata location to /mnt/disk1....rebuilt everything....lasted for not even 28 hours before I was back to corrupted DBs.  I deleted everything related to Plex Docker, started over using binhex-PlexPass, set appdata location to /mnt/disk1....corrupted again, but lasted for 2 days this time.  Back to the drawing board, same set up with binhex-PlexPass and location, BUT, returned my UNRAID to 6.6.7.   Everything is running stable so far now for 3 days. 

     

    Here's what I found when I went to the logs to see what was causing the issue

     

    First, everything was relatively stable until I started added media content to my library

     

    After a few loads, I would see that the media was being updated into Plex even though it was on the drive and in proper Plex format

     

    Next, when I went to my TV Shows, it would show zero seasons, but if you clicked on the thumbnail, to go to the individual season folders, they'd be there, go back to home and then try to get back, you couldn't!  It would show library as empty.

     

    If I went to movie folder and clicked on it, I would see the movies I had without the ones I just loaded.  When I went back to home, all the sudden my movie library showed as zero.

     

    In both cases, TV Shows and Movies, once they showed zero, then all the library folders showed zero content and I would get the gray screen of death.

     

    Looked at log files, and sure enough, in every corruption case, it shows Sqlite errors,

     

    Seems with what I'm seeing now, defining the appdata location to cache or diskX, and going back to unraid 6.6.7 is the way to go.  I just hope this gets fixed on the Sqlite side as I'm nervouse that this could only be a band-aid if Plex alters some of their coding.

     

    I hope this provides some more data to help fix the issue. 

    • Like 1
    Link to comment



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.