• Downgraded back to 6.6.7 due to Sqlite corruption


    limetech
    • Solved Urgent

    This is continuation of this original issue report.

     

    There is some conflicting information in that topic and, therefore, with your help, we want to try and get to the bottom of this.  We have several test servers running trying to reproduce this problem, so far hasn't happened.

     

    The first thing we want to definitively establish is whether the problem is isolated to only those cases where /config inside the container is mapped to:

    /mnt/user/appdata

    vs.

    /mnt/cache/appdata

    or

    /mnt/diskN/appdata

     

    Note: it does not matter what the "Default appdata storage location" is set to on Settings/Docker page.  Of interest is the actual path used by an individual container.

     

    What I'd like to ask is this:  For those of you who have seen this DB corruption occur after updating from Unraid 6.6.x to 6.7.x, please note the placement of your 'appdata' folder and how it's mapped to your problematic container.  Then update to 6.7.2 and run normally until you see corruption.  At that point, please capture diagnostics and report how appdata is being mapped.  If this is a huge PITA for you, then bow out of this testing - we are not trying to cause big disruptions for you.

     

    Finally I want to make clear: we are not "blaming" this issue on any other component, h/w or s/w, and also not "admitting" this is our problem.  We have not changed any code we're responsible for that would account for this, which doesn't mean we don't have a bug somewhere - probably we do, and a change in the kernel or some other component might simply be exposing some long hidden issue.  We are only interested in identifying and fixing the problem.

    • Upvote 4



    User Feedback

    Recommended Comments



    I have now had two Plex database corruptions in the last 30min with the second one being somewhat intentional. I am starting with a fairly new Plex appdata directory as my previous database backups became corrupted so I chose to start over fresh. Today I downloaded a new media file and copied it to /mnt/user/movies and shortly thereafter tried to play the new media file and which failed and I found database corruption. So I restored to my most recent backup and tried adding another new media file. This media file was automatically discovered by Plex during the copy process to /mnt/user/movies. No corruption occurred until I refreshed the Plex On Deck page which started to show the new media file and at this point I see corrupt database error messages in the Plex logs.

     

    My appdata share only resides on disk2. I do have a cache disk and have confirmed that no appdata directory is present on the cache drive. My Plex docker container is configured to use /mnt/disk2/appdata/plex. My system is still running 6.7.0.

     

    Edited to add, all non-cache disks are formatted XFS

     

    Edited by sjoerger
    Link to comment

    @limetech 

     

    I just managed to corrupt my sqllite db's on both Radarr and Sonarr (never checked Plex at the time it happened though).

     

    I have always used /mnt/user/appdata for everything, and I do not have any problems.  But, in the course of testing something out for another user here, I enabled DirectIO (was previously at auto).  With that enabled, the db's immediately crashed.  After I noticed the corruption, I reverted DirectIO back to "No", and everything fired up correctly.

     

    Hopefully this is a data point you can use in solving this problem.  Diagnostics attached which covers everything

     

    
    [v0.2.0.1344] NzbDrone.Core.Datastore.CorruptDatabaseException: Database file: /config/nzbdrone.db is corrupt, restore from backup if available. See: https://github.com/Radarr/Radarr/wiki/FAQ#i-am-getting-an-error-database-disk-image-is-malformed ---> System.Data.SQLite.SQLiteException: disk I/O error
    
    disk I/O error
    

     

    EDIT: Should actually clarify that no actual corruption happened.  SQLite thought it was corrupted if DirectIO was enabled.  With it disabled, everything came back alive with no issues

     

    servera-diagnostics-20190629-2304.zip

    Edited by Squid
    • Like 1
    Link to comment
    1 hour ago, Squid said:

    @limetech 

     

    I just managed to corrupt my sqllite db's on both Radarr and Sonarr (never checked Plex at the time it happened though).

     

    I have always used /mnt/user/appdata for everything, and I do not have any problems.  But, in the course of testing something out for another user here, I enabled DirectIO (was previously at auto).  With that enabled, the db's immediately crashed.  After I noticed the corruption, I reverted DirectIO back to "No", and everything fired up correctly.

     

    Hopefully this is a data point you can use in solving this problem.  Diagnostics attached which covers everything

     

    
    [v0.2.0.1344] NzbDrone.Core.Datastore.CorruptDatabaseException: Database file: /config/nzbdrone.db is corrupt, restore from backup if available. See: https://github.com/Radarr/Radarr/wiki/FAQ#i-am-getting-an-error-database-disk-image-is-malformed ---> System.Data.SQLite.SQLiteException: disk I/O error
    
    disk I/O error
    

     

    EDIT: Should actually clarify that no actual corruption happened.  SQLite thought it was corrupted if DirectIO was enabled.  With it disabled, everything came back alive with no issues

     

    servera-diagnostics-20190629-2304.zip 233.01 kB · 0 downloads

    I can attest to this - I forgot that I turned on DirectIO somewhere along the way..  I turned it off, restarted my array and all my docker containers now start properly even with the path set to /mnt/user/appdata. Turn it back on - none of my containers start - DB errors everywhere.

     

    Not saying this is necessarily related to db corruption over time issues that users are having - but in my case this is definitely the culprit.

     

    Saw this thread from a while ago with people having similar issues:

     

     

    Edited by nphil
    • Upvote 1
    Link to comment

    So I've been thinking. I haven't had any more problems with my Plex database. However, my usage has changed. I was experiencing corruption almost constantly when I was first setting up my server and loading it with new movies every day. Now that I have my movie collection digitized I'm not adding new movies very often. I also limited my Plex to a single movie library.

    I think it is time to add my Music library again and my paltry TV Show library and see if anything changes. My in-laws are moving in this week, so I will probably begin adding their movies to our server before long. I will report back if I have any more trouble in the coming weeks.

    @limetech, thank you for working with us on this issue!

    Link to comment

    I have had this issue for the past 2 months or so. Initially it affected only plex but it has started affecting sonarr and radarr as well. I seem to experience the corruption at a much greater frequency than others have reported. Often I will have to restore each database 2-3+ times per week. I have thoroughly read the other thread and implemented every precaution/fix listed there (appdata is set to disk1, plex update every hour, sonarr and radarr disconnected from plex, etc.) I spent this morning making sure all of the databases were up to date and uncorrupted, and then upgraded to unraid 6.7.2. Unfortunately not even 8 hours later, Plex is corrupted. I am extremely motivated to help find a fix for this issue. I have attached my server diagnostics and I am willing to spend time testing/providing any other logs necessary. Thank you for taking the time to look into this.

    serenity-diagnostics-20190702-0021.zip

    Link to comment

    When I bought my unraid pro key on 5/25, 6.7.0 was the current version; that's what I installed then, and it's what I'm running now.  My system has never had another version on it.  It's running a clean install of unraid 6.7.0 from the late May iso.  I initially had my containers running out of /mnt/user/media/appdata (the user share) and soon had sqlite corruption in all my docker containers (plex official, binhex radarr and sonarr).  Reading a forum post here led me to try moving them to /mnt/disk1/media/appdata (a member disk in the user share), but I had sqlite corruption there too.  I'd delete and clean install containers, and in under a day, the db's would be corrupt.

    So I did something outside the box; I picked up a 128GB usb 3.1 thumb drive, formatted it xfs, plugged it into my unraid box, mounted it, and moved all my containers to it.  Since then, I've had no issues with sqlite corruption.  The setup is far from ideal, but I've been running like this for 2 weeks now, and not seen this problem.  The thumbdrive currently shows up under 'Unassigned Devices', and is identified as "

    Samsung_Flash_Drive_FIT_0352819030000266-0:0 - 128 GB (sdb)".  Hopefully that helps narrow down the problem.

    Link to comment
    10 hours ago, etcaz said:

    When I bought my unraid pro key on 5/25, 6.7.0 was the current version; that's what I installed then, and it's what I'm running now.  My system has never had another version on it.  It's running a clean install of unraid 6.7.0 from the late May iso.  I initially had my containers running out of /mnt/user/media/appdata (the user share) and soon had sqlite corruption in all my docker containers (plex official, binhex radarr and sonarr).  Reading a forum post here led me to try moving them to /mnt/disk1/media/appdata (a member disk in the user share), but I had sqlite corruption there too.  I'd delete and clean install containers, and in under a day, the db's would be corrupt.

    So I did something outside the box; I picked up a 128GB usb 3.1 thumb drive, formatted it xfs, plugged it into my unraid box, mounted it, and moved all my containers to it.  Since then, I've had no issues with sqlite corruption.  The setup is far from ideal, but I've been running like this for 2 weeks now, and not seen this problem.  The thumbdrive currently shows up under 'Unassigned Devices', and is identified as "

    Samsung_Flash_Drive_FIT_0352819030000266-0:0 - 128 GB (sdb)".  Hopefully that helps narrow down the problem.

     

    You mean that you moved the docker.img to the USB or just the appdata?

    Link to comment

    I'm having issues with Plex sqlite db corruption after upgrade to 6.7.1 which happened almost every hour. My docker is set to point to /mnt/user/appdata

    I downgraded to 6.6.6 and the issue went away. The same restored sqlite db that had corrupted within an hour on 6.7.1 hasn't corrupted yet on 6.6.6 after weeks of usage.

    Link to comment
    On 7/2/2019 at 9:28 AM, TXLZONE said:

    I'm having issues with Plex sqlite db corruption after upgrade to 6.7.1 which happened almost every hour. My docker is set to point to /mnt/user/appdata

    I downgraded to 6.6.6 and the issue went away. The same restored sqlite db that had corrupted within an hour on 6.7.1 hasn't corrupted yet on 6.6.6 after weeks of usage.

    I am curious, what setting did you have set on DirectIO option, when you were having corruption issues?

     

    Link to comment
    45 minutes ago, semtex41 said:

    I am curious, what setting did you have set on DirectIO option, when you were having corruption issues?

     

    When I checked it I remember it was in the default setting which I believe is "Auto". If you can refresh my memory on what the screen looks like in version 7.x, I can confirm 100% what it was set to.

    • Upvote 1
    Link to comment

    @ps2sunvalley often the UI wont load, or some sections of the UI won't load.

     

    You can test SQLite databases with this command... replace it with the real database name...

     

    sqlite3 database_name.db "PRAGMA integrity_check"

     

    Link to comment

    Like most, I have had constant db corruption since 6.7. (Plex, Sonarr, & Radarr)

     

    - Sunday, corrupted while on v6.7.1 & Config mapped to /mnt/user/appdata

    - Sunday, restored backup db's, changed Config mapping to /mnt/diskN/appdata, and updated to v6.7.2

    - Today, db's corrupted again. (They may have been corrupted for a few days, but only noticed today)  Diagnostics report attached

     

     

    Update:

    - db corrupted almost instantly after backup restore and import of 4 new files

    - I have now disabled DirectIO and restored backups...again. Will update if corrupts again.

     

    tower-diagnostics-20190703-2254.zip

    Edited by MrFlood
    More info
    Link to comment

    Is anyone else having issues changing the mapping to the appdata?

     

    I've been using /mnt/user/appdata/PlexMediaServer/ for 1.5 years.  If I try to only change the mapping in the docker config to /mnt/cache/appdata/PlexMediaServer/ I get this error on an infinite loop:

     

     

     

    ****** PLEX MEDIA SERVER CRASHED, CRASH REPORT WRITTEN: /config/Library/Application Support/Plex Media Server/Crash Reports/1.16.2.1297-4b7ace214/PLEX MEDIA SERVER/55e3a89d-90b5-b4b2-3afbf75b-4e9ba94f.dmp
    Starting Plex Media Server.
    terminate called after throwing an instance of 'std::runtime_error'


    And then when I change the directory back to /mnt/user/appdata/PlexMediaServer/ everything returns to normal, and eventually I may get an SQLite error (haven't seen it in a few days now).

     

    I'm on 6.7.2 right now.

    Link to comment
    6 hours ago, ps2sunvalley said:

    Is anyone else having issues changing the mapping to the appdata?

     

    Is the appdata share completely on the cache drive?  Diagnostics would show, but also under the shares tab click on Compute next to appdata

    Link to comment
    15 hours ago, Squid said:

    Is the appdata share completely on the cache drive?  Diagnostics would show, but also under the shares tab click on Compute next to appdata

    Yep, appdata only exists on the cache drive.

    Link to comment

    Got hit by a triple whammy and had Radarr, Sonarr, and Ombi all get their databases corrupted yesterday. Everything was fine in the afternoon but by the evening they were all dead.

    They were using the same mounting as mentioned in my previous message (/mnt/user/appdata/{sonarr|radarr|ombi}). The appdata share is split at the top level so the entire sonarr|radarr|ombi would be self contained on a single disk.

    I decided to complete remake the containers and their associated config folders however this time I am mounting directly using a disk share. So now the config folders are /mnt/disk1/appdata/{sonarr|radarr|ombi}. If I get any corruption with this configuration I'll capture diagnostics again and upload here.

    Side questions:
    1. Using one of the disk mounts (ie /mnt/disk1/appdata/sonarr) instead of a user mount (/mnt/user/appdata/sonarr), is that data still protected by the parity disk?
    2. Is it ok to access those folders using an SMB share via windows? (the appdata share is available over SMB, is it safe to access the appdata/sonarr folder if the container has mounted /mnt/disk1/appdata/sonarr)

    Edited by mdeabreu
    Link to comment

    1. Yes.

    2. Yes. Just never do a copy involving /mnt/disk# and /mnt/user unless you know exactly what you're doing (see user share file copy issue that could overwrite a file with itself and ends up at 0 bytes).

    Link to comment
    On 6/30/2019 at 3:58 PM, Brian H. said:

    So I've been thinking. I haven't had any more problems with my Plex database.

    Oops I've been running 6.6.7 for a while without realizing it! I thought about doing it. But I don't remember pulling the trigger.

    I suspect that is why I haven't been having corruption issues. I just made a backup of my Plex database and updated to 6.7.2. Now is a perfect time for me to test again as I just begun the process of ripping my in-laws movie collection.

    I switch to running the linuxserver Plex docker, instead of binhex's (I realized I needed to run that one so I could actually enable NVIDIA hardware encoding.) And I am going to try setting its app data path to /mnt/disk1/appdata/plex.

    Link to comment

    Update from last post - DB has corrupted AGAIN! both Sonarr and Radarr.

     

    Config Reminder

    - V6.7.2

    - mapping was set to /mnt/diskN/appdata

    - DirectIO was disabled

     

     

    How do I downagrade back to 6.6.7 and will it fix the issue?

     

     

     

     

    Link to comment



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.