• Downgraded back to 6.6.7 due to Sqlite corruption


    limetech
    • Solved Urgent

    This is continuation of this original issue report.

     

    There is some conflicting information in that topic and, therefore, with your help, we want to try and get to the bottom of this.  We have several test servers running trying to reproduce this problem, so far hasn't happened.

     

    The first thing we want to definitively establish is whether the problem is isolated to only those cases where /config inside the container is mapped to:

    /mnt/user/appdata

    vs.

    /mnt/cache/appdata

    or

    /mnt/diskN/appdata

     

    Note: it does not matter what the "Default appdata storage location" is set to on Settings/Docker page.  Of interest is the actual path used by an individual container.

     

    What I'd like to ask is this:  For those of you who have seen this DB corruption occur after updating from Unraid 6.6.x to 6.7.x, please note the placement of your 'appdata' folder and how it's mapped to your problematic container.  Then update to 6.7.2 and run normally until you see corruption.  At that point, please capture diagnostics and report how appdata is being mapped.  If this is a huge PITA for you, then bow out of this testing - we are not trying to cause big disruptions for you.

     

    Finally I want to make clear: we are not "blaming" this issue on any other component, h/w or s/w, and also not "admitting" this is our problem.  We have not changed any code we're responsible for that would account for this, which doesn't mean we don't have a bug somewhere - probably we do, and a change in the kernel or some other component might simply be exposing some long hidden issue.  We are only interested in identifying and fixing the problem.

    • Upvote 4



    User Feedback

    Recommended Comments



    I had the corruption in /mnt/user/appdata/ and /mnt/disk3/appdata in plex, sonarr, and tautulli.  Corruption happened most when writing to the sqlite DB as seen through automatic scans when adding new media from sonarr.  Or, delete the metadata and scan the media again and it would corrupt.

     

    Right now I'm on /mnt/cache/appdata and haven't seen the same corruption. That's been going solid for the last 2 or 3 days. I'm on 6.7.1.

     

    If it would help, I can open my server up to you guys to SSH in for testing.

    Edited by runraid
    Link to comment
    25 minutes ago, runraid said:

     

    I had the corruption in /mnt/user/appdata/ and /mnt/disk3/appdata

    This corruption occurred without any intervening system crash, correct?  What file system is on /mnt/disk3?

     

    Are you absolutely double-dog positive you saw corruption with appdata existing solely on /mnt/disk3 and all /config

    paths were set to /mnt/disk3/appdata (and nothing sneaked by as /mnt/user/appdata)?

     

    Also please post diags so that we can start collecting h/w information for systems that experience corruption.

    Link to comment

    I did not have a system crash. And I’m absolutely positive I had corruption on /mnt/disk3 but the rate at which it occurred was slowed. It’s the xfs fs. Here’s my diags. 

    tower-diagnostics-20190625-2050.zip

     

    Update: I'm now on 6.7.2. The diagnostics was from 6.7.1

    Edited by runraid
    Link to comment

    I am on 6.7.1 and db corruption continues. /mnt/user/appdata and /mnt/disk1/appdata have both suffered corruption.  I have no cache disk (out of sata ports until my lsi card gets here).

     

    xfs

     

    no system crash, just running as expected.

     

    This happens frequently in Sonarr/Radarr and Plex

     

    I will update to 6.7.2 to test. 

     

    not sure if it's anyway related.. But I'm seeing tons of "Waited one whole second for a busy database." in the plex console. Not sure if it was like this previously during an import of a library.

     

     

    Update: It happened sometime overnight, I had a few movies queued in in Radarr but it appears those completed. It corrupted sometime after.

    server-diagnostics-20190626-1058.zip

    Edited by spyd4r
    • Upvote 1
    Link to comment

    The Plex database corrupted twice on my server, and Sonarr at least twice, resulting in many of my hours used resolving the problems.  Both were stored in /mnt/user/appdata/.  I have moved everything to /mnt/cache/appdata/ by toggling the 'use cache only' setting and so far no database corruptions.  I appreciate limetech taking this matter seriously and want to say thanks for an otherwise fantastic 6.7.0 release.

    • Like 2
    • Upvote 1
    Link to comment

    My appdata was on /mnt/user/appdata. I experienced corruption (Plex sqlite db) after upgrading to 6.7.0. After a reinstall of Plex, it corrupted after a few hours. I downgraded back to 6.6.7 which solved the issue. After reading and commenting on the previous thread, I changed my appdata to /mnt/disk1/appdata, upgraded to 6.7.0 and reinstalled Plex. I have not experienced corruption since.

     

    I have no time to test but maybe this information aids in solving this. :)

    unraid-diagnostics-20190626-0903.zip

    Link to comment

    Question: Why it only happens for users using Plex/Sonar/Radar? These are the 3 I always read. I use none of them and don't had any issues so far.

    Link to comment
    2 hours ago, bastl said:

    Question: Why it only happens for users using Plex/Sonar/Radar? These are the 3 I always read. I use none of them and don't had any issues so far.

    Maybe because you didn't read everything.

    I only have Plex, nothing else...

    On vacation now but will get on this once I get back. Can also provide remote to devs if needed. I seem to be able to crash this fairly quickly. From what I recall it usually crashes when Plex is doing a scan and I move a number of new files in at the same time. 

    Link to comment

    Dear Unraid team,

    thank you for looking into this. I have had Plex corruptions since upgrading so I downgraded.

    Unfortunately I cannot run the test as I do not have the time for this and do not want to take any risk with my data, even though it is backed up.

    But for what is worth and FYI, my app data path is as follow: /mnt/user/appdata

    If you need any log file, etc. please let me know.

    Link to comment
    3 hours ago, bastl said:

    Question: Why it only happens for users using Plex/Sonar/Radar? These are the 3 I always read. I use none of them and don't had any issues so far.

    Maybe because other Dockers use SQLite differently from other runtimes, like if Emby uses it then its through the dotNet Framework and not from C++ or PHP?

    Link to comment

    I have upgraded to 6.7.2.  

     

    Both of my dockers are set to /mnt/user/appdata.  

     

    I will check on things tonight and see how it is going.  

     

     

    Screen Shot 2019-06-26 at 7.29.38 AM.png

    Link to comment

    Currently running 6.7.2. Started seeing this issue after upgrading to 6.7.0 and then rolled back to 6.5.1. However, I still saw the issue occasionally on the older version. I'm not sure which docker is crashing - I run Plex, Radarr, Sonarr, and binhex-Sonarr and others . Every 1-3 days I see 100% CPU load and dockers not responding and I need to restart the array. I think it is probably Plex that crashes based on a Reddit thread I saw. From Reddit I understand that most people are able to role back to 6.6.x and not see the issue, so the fact that I still saw the issue after rolling back to 6.5.1 is surprising to me given how similar the symptoms of my issue are to this thread. 

     

    All of my /config paths map to /mnt/user/appdata .

     

    I noticed binhex-sonarr has a /data path mapped to /mnt/cache/appdata/data. Seems like a bad idea. I'm not sure what that path is for, the other Sonarr docker doesn't have it and it's my only docker mapping pointing at the cache drive. I've stopped the docker for now to see if it helps. 

     

    I'll capture the diagnostics when I hit the issue on 6.7.2, I just updated.

    Link to comment
    18 minutes ago, RadBrad said:

    Every 1-3 days I see 100% CPU load and dockers not responding and I need to restart the array

    I think this is an unrelated Plex issue. I sorted this by rolling Plex back to 1.14.1.5488-1-01

    Link to comment

    Some background:

     

    I was running 6.7.0 and eveyrthing was fine. Upgraded to 6.7.1 - any container that had a .db file stored in the cache drive refused to start. Changed path of the docker containers from /mnt/user/appdata to /mnt/cache/appdata and everything has been running smoothly for almost 3 days now.

     

    Just for a test, I upgraded to 6.7.2 and installed a bitwarden container with everything default and the db mapped to /mnt/user/appdata. I keep getting DB errors right away and the container refuses to start. Switching the path to /mnt/cache/appdata resolves the issue immediately.

     

    So in my case, it's not a matter of  DB corruption over time ( at least not that I've noticed yet) - docker containers that have db files with the appdata path set as /mnt/user/appdata just refuse to start unless changed to /mnt/cache/appdata.

     

     

    beastnas-diagnostics-20190626-2148.zip

    Edited by nphil
    Link to comment
    29 minutes ago, nphil said:

    So in my case, it's not a matter of  DB corruption over time ( at least not that I've noticed yet) - docker containers that have db files with the appdata path set as /mnt/user/appdata just refuse to start unless changed to /mnt/cache/appdata.

     

    Thank you for the report.  If willing please try this:

    1. Assuming you are running 6.7.2.

    2. Copy the attached file to the 'config' folder on your USB flash device.  You can do this by copying to 'flash' share on your network (into 'config' folder), or plug flash into PC, or create the file via console - it just has a single line in there that reads:

    shfsExtra=-o use_ino

    3. Restore the container config path back to /mnt/user/appdata

    4. Now Stop/Start array for setting to take effect and see if you get same DB corruption.

     

    If the application is relying on hard links then this should fix it.  However, with this option in effect, certain older media players and DVD players will have problems accessing media via NFS.

     

    To restore normal operation, delete or rename that file and then Stop/Start array.

     

    Edit: an easy way to create the file is open Terminal window and then select/copy/paste this and hit return:

    echo "shfsExtra=-o use_ino" >/boot/config/extra.cfg

    extra.cfg

    Link to comment

    Ok.  I upgraded this morning to 6.7.2.  Both of my dockers were set to /mnt/user/appdata.  

     

    Screen Shot 2019-06-26 at 7.29.38 AM.png

     

    I did not use Plex at all today, and no downloads from Sonnar.  Plex was doing it normal cleanup, looking for captions, etc.  

     

    The database corrupted.  

     

    Attached are the diags.   

     

    MB:  Azrock Z97M Pro4.  AMI bios version P2.10 May 27, 2016.  There is one bios update, but AZROCK says it is only Hazwell firmware updates.  

     

     

    swissarmy-diagnostics-20190627-0056.zip

    Link to comment
    4 hours ago, limetech said:

     

    Thank you for the report.  If willing please try this:

    1. Assuming you are running 6.7.2.

    2. Copy the attached file to the 'config' folder on your USB flash device.  You can do this by copying to 'flash' share on your network (into 'config' folder), or plug flash into PC, or create the file via console - it just has a single line in there that reads:

    
    shfsExtra=-o use_ino

    3. Restore the container config path back to /mnt/user/appdata

    4. Now Stop/Start array for setting to take effect and see if you get same DB corruption.

     

    If the application is relying on hard links then this should fix it.  However, with this option in effect, certain older media players and DVD players will have problems accessing media via NFS.

     

    To restore normal operation, delete or rename that file and then Stop/Start array.

     

    Edit: an easy way to create the file is open Terminal window and then select/copy/paste this and hit return:

    
    echo "shfsExtra=-o use_ino" >/boot/config/extra.cfg

    extra.cfg 21 B · 0 downloads

     

    Thanks for looking into this. Unfortunately, this didn't seem to make any difference. I verified the extra config file and started/stopped the array twice to double check.  Multiple docker containers (plex, sonarr, radarr, nginxproxy, nzbhydra, bitwarden etc.) refuse to start unless I switch back to /mnt/cache/appdata. They all give some variation of "database corrupted" error.

     

    What I don't understand is why I didn't have any issues with 6.7.0 (where these first user reports started popping up), and only after I upgraded to 6.7.1.

     

    Let me know if there are some other steps I can take.

    Edited by nphil
    Link to comment

    thanks to the team for trying to resolve this issue.

     

    I am also effected by this issue. I also noticed it first happen upon upgrading to 6.7.0.. Sonarr was effected, path to /mnt/user/appdata

     

    nzbHydra v2 also seems to corrupt after a day and i cant open webui, restarting the docker doesn't help. path also to /mnt/user/appdata

     

    SQL State : 90030
    Error Code : 90030

     

    Please advise what steps to take if you need anything from my system.

     

    thanks

    Link to comment

    FYI. The just released version of plex now supports push notifications. It also started monitoring the database for corruption and will send you a push notification if the database becomes corrupted. 

    Link to comment

    Thanks for looking into this. I've been bitten by it a couple of times now. Upgraded to 6.7.2 recently and things were running fine. I tried adding some media to Plex and it corrupted the database.
     

    Mounted using /mnt/user/appdata

    Diagnostics attached.

    plexmappings.JPG

    tower-diagnostics-20190628-1658.zip

    Link to comment
    14 minutes ago, mdeabreu said:

    Thanks for looking into this. I've been bitten by it a couple of times now. Upgraded to 6.7.2 recently and things were running fine. I tried adding some media to Plex and it corrupted the database.
     

    Mounted using /mnt/user/appdata

    Diagnostics attached.

    plexmappings.JPG

    tower-diagnostics-20190628-1658.zip 87.89 kB · 0 downloads

     

    Thank you for the report.  From your diags, looks like the 'appdata' share exists on both disk1 and disk2 and split-level is set to 1 (this is good).  Please verify for me that the 'appdata/plex' directory only exists either on disk1 or disk2 (or let me know if it exists on both disks).

    Link to comment
    3 minutes ago, limetech said:

     

    Thank you for the report.  From your diags, looks like the 'appdata' share exists on both disk1 and disk2 and split-level is set to 1 (this is good).  Please verify for me that the 'appdata/plex' directory only exists either on disk1 or disk2 (or let me know if it exists on both disks).

    `appdata/plex` is indeed solely on disk2.

    I recovered the database from backups and attempted to re-add the media (believing rapid db access to be related) however it was able to successfully add the media without issue.

    Link to comment
    3 minutes ago, mdeabreu said:

    `appdata/plex` is indeed solely on disk2.

    I recovered the database from backups and attempted to re-add the media (believing rapid db access to be related) however it was able to successfully add the media without issue.

    This is indeed a maddening problem 🤬

     

    If you see corruption again, please recover again from backup and then change your Plex /config mapping to:

    /mnt/disk2/appdata/plex

     

    If possible don't change any other config setting or add/remove devices.

    Link to comment
    On 6/26/2019 at 8:10 PM, Rich Minear said:

    Ok.  I upgraded this morning to 6.7.2.  Both of my dockers were set to /mnt/user/appdata.  

     

    Screen Shot 2019-06-26 at 7.29.38 AM.png

     

    I did not use Plex at all today, and no downloads from Sonnar.  Plex was doing it normal cleanup, looking for captions, etc.  

     

    The database corrupted.  

     

    Attached are the diags.   

     

    MB:  Azrock Z97M Pro4.  AMI bios version P2.10 May 27, 2016.  There is one bios update, but AZROCK says it is only Hazwell firmware updates.  

     

     

    swissarmy-diagnostics-20190627-0056.zip 112.62 kB · 2 downloads

    @limetech Is there anything I should be doing different based on my diagnostics?  Anything you would like for me to try?

    Link to comment



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.