Jump to content
Rich Minear

Downgraded back to 6.6.7 due to Sqlite corruption

192 posts in this topic Last Reply

Recommended Posts

@Brian H. 6.7.1-rc2 was released. In the release notes I see

 

> shfs: support FUSE use_ino option

 

No idea what that enables though.

Share this post


Link to post
Posted (edited)
9 hours ago, runraid said:

@Brian H. 6.7.1-rc2 was released. In the release notes I see

 

> shfs: support FUSE use_ino option

 

No idea what that enables though.

so out of the Debian man pages:

 

use_ino

Honor the st_ino field in kernel functions getattr() and fill_dir(). This value is used to fill in the st_ino field in the stat(2), lstat(2), fstat(2) functions and the d_ino field in the readdir(2) function. The filesystem does not have to guarantee uniqueness, however some applications rely on this value being unique for the whole filesystem.

 

if the inode numbers are not being honored unique currently, it might piss off sqlite.  This sounds worth pursuing.  I'm not super familiar with fuse, but it's supposed to use an internal mechanism instead of inodes, but if an app is unaware or expects inodes, then I could see it causing the file to become malformed. 

 

This would also explain why its better when kept further from fuse shares (like /mnt/user).

 

I am on the list too of having problems.  I currently have added cache and have changed appdata/system it to cache only and have the plex container pointing there... maybe that will help, only been 6 hours since the last corruption. 

 

Honestly I couldn't give a f about rescanning my library, it's annoying, but the dvr schedule, stuff programmed to be recorded that is then lost, etc, is really getting old.  really thinking about just putting omv back on here.

Edited by Abzstrak

Share this post


Link to post
On 6/15/2019 at 3:10 AM, CHBMB said:

One thing that does stand out, it seems that everyone is changing their disks and restoring old databases, I'm curious if starting with a fresh database would fix this.

The reason I say this, is historically only people who use /mnt/user/ have ever been affected by this issue. This still seems to be the case, but for whatever reason, the issue seems to be affecting more people now, but it doesn't seem to be affecting anyone new who was already using /mnt/cache/ or /mnt/disk1/ but those that have changed from /mnt/user/ to /mnt/cache/ it doesn't seem to resolve it, so I think we need to find out if this is still the case with a fresh install.

I know that's a bit of a PITA, but it's a question worth asking.

My other thought, is I don't think it's entirely fair to dump this solely at LimeTech's feet, yes they are the distro maintainer, but that doesn't necessarily mean that every issue is their fault. This could be a weird quirk of a particular hardware combo, as it's never been possible to isolate why this has affected some, but not all, users.

Let's keep the drama down, do some troubleshooting and try and isolate the issue.

Sent from my Mi A1 using Tapatalk
 

No.  I've tried the new database.  I restored two copies before with both Sonar and Plex.  The last time, I went the route of creating a new database with Plex.  It still corrupted.  That is when I downgraded Unraid.  

Share this post


Link to post

Just came across this post and wanted to add another to the list. Plex has been running solid since my last server rebuild nearly 2.5 years ago. I upgraded to 6.7 a few days after release and a week or two later Plex became corrupt. If I recall it was shortly after an update to the Plex container...so I never suspected it as a potential Unraid issue. Since Plex had been running for so long, and I was kind of wanting to change some of my libraries around, I decided to just do a rebuild instead of restoring from backups.

 

I also run Sonarr/Radarr/Lidarr and no issues....but they are all using /mnt/cache. I would have sworn the same was for Plex, but I just pulled down a backup prior to my rebuild and sure enough, I used /mnt/user. It was the first container I setup and probably didn't know any better at the time.

 

I believe someone also mentioned something about CA Backup / Restore Appdata...I also run weekly backups of all containers...but do not recall if corruption happened after a backup. Even if it did this seems more like a possible correlation, but not the underlying cause.

 

I know this is all anecdotal, but @CHBMB may be on to something with not only switching to /mnt/cache or disk, but also creating a clean install with clean database. So far, I've had no issues with Plex since my clean install almost three weeks ago.

Share this post


Link to post
3 minutes ago, Rich Minear said:

No.  I've tried the new database.  I restored two copies before with both Sonar and Plex.  The last time, I went the route of creating a new database with Plex.  It still corrupted.  That is when I downgraded Unraid.  

I have too.  I started out with unraid 6.7.0, i've only been using it a couple of weeks, still on the trial.... not impressed and it was a new install, new database, etc...  first got a db corruption to the point plex wouldn't start within 3 days of the install.   I came from using OMV where Plex ran for the last 18 months and I'd never seen a corrupt database.

Share this post


Link to post
1 minute ago, Spatial Disorder said:

Just came across this post and wanted to add another to the list. Plex has been running solid since my last server rebuild nearly 2.5 years ago. I upgraded to 6.7 a few days after release and a week or two later Plex became corrupt. If I recall it was shortly after an update to the Plex container...so I never suspected it as a potential Unraid issue. Since Plex had been running for so long, and I was kind of wanting to change some of my libraries around, I decided to just do a rebuild instead of restoring from backups.

 

I also run Sonarr/Radarr/Lidarr and no issues....but they are all using /mnt/cache. I would have sworn the same was for Plex, but I just pulled down a backup prior to my rebuild and sure enough, I used /mnt/user. It was the first container I setup and probably didn't know any better at the time.

 

I believe someone also mentioned something about CA Backup / Restore Appdata...I also run weekly backups of all containers...but do not recall if corruption happened after a backup. Even if it did this seems more like a possible correlation, but not the underlying cause.

 

I know this is all anecdotal, but @CHBMB may be on to something with not only switching to /mnt/cache or disk, but also creating a clean install with clean database. So far, I've had no issues with Plex since my clean install almost three weeks ago.

while plex is the most prevalent, people are seeing with with most apps that use sqlite.  I'm still on the trial and have little desire to spend $130 on a piece of software that doesn't work for my purpose.  I figured unraid would be easy and allow some extra power savings by spinning down drives, but I've been a linux system admin for 15 years, I'm strongly thinking of dumping it for a normal distro or going back to omv on the box. 

 

It's also highly irritating that this seems to be fairly wide spread, but not being addressed by limetech.

Share this post


Link to post
32 minutes ago, Abzstrak said:

It's also highly irritating that this seems to be fairly wide spread, but not being addressed by limetech.

I do not think we know this to be the case !    Limetech could be trying feverishly to identify the cause and therefore how to fix it and we would not necessarily know about it.

Share this post


Link to post
2 minutes ago, itimpi said:

I do not think we know this to be the case !    Limetech could be trying feverishly to identify the cause and therefore how to fix it and we would not necessarily know about it.

 

Sorry, I might not have been clear... I meant address it, like mention they are looking into it or something... I don't expect anyone to pull a fix out of thin air, but addressing the situation publicly should be done.  Even if its addressing it and saying they don't think its real, at least communicate about it.

Share this post


Link to post
15 minutes ago, Abzstrak said:

I meant address it, like mention they are looking into it or something...

 

On 6/5/2019 at 2:26 PM, limetech said:

We have been looking into this for the past few days.  If you have a way to make this happen on demand, please post how you do it.

 

On 6/15/2019 at 2:12 AM, eschultz said:

Has anyone here tried Unraid 6.7.1-rc2 release to see if the sqlite corruption issues are still there?

 

Share this post


Link to post
Posted (edited)

It just occurred to me that I should have tagged him on my last post where I shared more information about my hardware and some of the things I have done to troubleshoot it. I tagged him in that post. Maybe it can help.

Edited by Brian H.
Removing repetitive defense of @limetech. ;)

Share this post


Link to post

@runraid@Abzstrak:

I just installed unRaid 6.7.1-rc2. However, binhex-plexpass would not run due to a "bad parameter". If I removed --runtime=nvidia from "Extra Parameters" I was able to get it running. I haven't used hardware acceleration yet so it is not a problem for now. It is worth a try.

Share this post


Link to post
1 hour ago, Abzstrak said:

while plex is the most prevalent, people are seeing with with most apps that use sqlite.  I'm still on the trial and have little desire to spend $130 on a piece of software that doesn't work for my purpose.  I figured unraid would be easy and allow some extra power savings by spinning down drives, but I've been a linux system admin for 15 years, I'm strongly thinking of dumping it for a normal distro or going back to omv on the box. 

 

It's also highly irritating that this seems to be fairly wide spread, but not being addressed by limetech.

I can appreciate that...especially when trying to decide which OS to go with. I will say, I jumped to Unraid about 2.5 years ago and it has been fantastic and one of the easiest to administer. The few times I've had issues this community has been extremely quick in helping resolve. Hell, I've gotten better support here than I have for enterprise products I pay six figures for (I'm an IT Manager).

 

I think with this particular issue it seems extremely hard to replicate. Plex corrupted on me but I'm also running lidarr, sonarr, radarr, bazarr, tautulli, syncthing, and pihole which all seem to be using sqlite to some extent...and I've no issues with those. Now that I took a closer look, pihole and syncthing are also using /mnt/user, instead of /mnt/cache, and they're fine so far (moving them now thought). So why did Plex corrupt and not the others?

Share this post


Link to post
1 hour ago, Spatial Disorder said:

I can appreciate that...especially when trying to decide which OS to go with. I will say, I jumped to Unraid about 2.5 years ago and it has been fantastic and one of the easiest to administer. The few times I've had issues this community has been extremely quick in helping resolve. Hell, I've gotten better support here than I have for enterprise products I pay six figures for (I'm an IT Manager).

 

I think with this particular issue it seems extremely hard to replicate. Plex corrupted on me but I'm also running lidarr, sonarr, radarr, bazarr, tautulli, syncthing, and pihole which all seem to be using sqlite to some extent...and I've no issues with those. Now that I took a closer look, pihole and syncthing are also using /mnt/user, instead of /mnt/cache, and they're fine so far (moving them now thought). So why did Plex corrupt and not the others?

So I had the corruption with both Sonarr and Plex.  Sqlite has been the issue....

Share this post


Link to post

For me the best way to replicate a database corruption is to have plex dvr recording 3 or more things simultaneously.... problem is thats still not 100%.  Seems more often than not surrounding dvr recordings, but that might be coincidence.

@Squid - I had missed that limetech had addressed this some, that's good, thank you.

 

@Spatial Disorder - I am running syncthing too, I haven't seen an issue with it, but I don't think its sqlite, is it?

Share this post


Link to post

All of you that have this as a frequent reocurring issue, could you check Settings, Global Share Settings, Tunable (enable Direct IO): and set it to No instead of Auto or Yes, and see if that changes anything?

Share this post


Link to post

Ffs. I’m on 6.7.1-rc2 and my plex just corrupted. I’m losing my mind with this. I’m not using fuse. And no, this is not getting the attention it needs. This is data corruption for a storage system.  We are paying customers and it seems to me like there’s hostility towards us for wanting this to get more attention.  I’m on the verge of going my own way without unraid. I can’t be dealing with corrupting data every couple of days. I’m will to give $200 to the unraid team to help prioritize this. Maybe if we all donate money we can make this a compelling reason to get this fixed. 

Share this post


Link to post
17 minutes ago, runraid said:

We are paying customers and it seems to me like there’s hostility towards us for wanting this to get more attention.

Honestly, I don't see how you'd figure that.  Someone posted that LT seems to be ignoring the issue, so a bunch of us replied stating that they are.

 

The big problem as far as I can see (I do not speak for limetech), is that the issue affects a minority of users for some reason.  And that makes it very difficult to determine what the underlying cause is if you cannot replicate the problem.

Share this post


Link to post

This certainly has all the attention of Limetech, and if a fix is available it will be presented as soon as possible.

This is not an easy problem to solve, I wish it was as simple as offering money to make a correction ...

 

Share this post


Link to post

I’ve set direct io to “no”. Once I get home I’ll repair my db and let you know if it corrupts again.   

C4A068FC-44B1-42E1-BE17-79B78295244C.jpeg

Share this post


Link to post

I'm on 6.7.0 with no cache drive, 4 x 4TB drives. I'm getting corrupt databases every couple of days. I rebuilt Radarr, Sonarr, and Plex this morning from scratch and manually re-configured them all. Sonarr and Plex are already corrupt 10 hours later. Going to downgrade to 6.6.x and see how things go (I didn't have any problems until a couple of days after I upgraded to 6.7.0).

Share this post


Link to post

Iam new on Unraid so I had a new fresh installation with version 6.7.0 and Plex is working fine.

So is it may an update failure?
What if these people's with having these struggles starting a fresh installation?
Would this help?
Just some thoughts of a newbie..

Sent from my MI 6 using Tapatalk

Share this post


Link to post
Posted (edited)
3 hours ago, jonathanm said:

All of you that have this as a frequent reocurring issue, could you check Settings, Global Share Settings, Tunable (enable Direct IO): and set it to No instead of Auto or Yes, and see if that changes anything?

Mine is set to auto (which is the same as No AFAIK), but I'll change it to No as soon as I can stop the array.

 

@Toobie, I'm new to unraid too, the first version I've ever installed is 6.7.0, and it has these corruption issues.... so no update was needed to cause it.

 

 

[edit]

I've set directIO to no now.

[/edit]

Edited by Abzstrak

Share this post


Link to post
5 hours ago, jonathanm said:

All of you that have this as a frequent reocurring issue, could you check Settings, Global Share Settings, Tunable (enable Direct IO): and set it to No instead of Auto or Yes, and see if that changes anything?

I will give that a try, thank you for the suggestion.

Share this post


Link to post
Posted (edited)

I have also been battling this issue - If I was to create a clean plex database, would it still know what episode I'm up to etc? 

 

My Sonarr corrupted again this morning with the share: Does fuse only get used if its in usr or if its a share full stop? 

/mnt/disk1/appdata_d1/Sonarr/

I don't have a cache drive but I do have a /mnt/cache directory in the system.. I don't know what this is to be completely honest because its only 6gb and I can't see the files I put in there on my flash drive.. where is this lol

 

Anyway I have moved a copy of my sonarr to there for now and will see what happens. Plex is running strong so far, but not much has been added to it in the last few days

Plex is using

/mnt/disk1/appdata_d1/Plex/

I was stable software wise on 6.6.x but was getting hangcheck error which I don't seem to get on 6.7 so I'm kinda stuck between crashing unraid and corrupting databases - kinda makes me not want to stay updated and find a version that runs and leave it there

 

Anyway, I've restored a database on sonarr and moved it to the cache direction as said above.. will update if it fails again which I expect it might

 

Other info that may be relevant

i7-2600

ASRock Z77 Pro3

12GB DDR3

 

running Sonarr/Plex/Jacket/ZoneMinder/DelugeVPN - only Sonarr and Plex have failed so far, plex hasn't failed since going to the disk1/appdata I created, temporarily, sonarr has 

 

Plugins installed:

CA Cleanup Appdata

OpenVPN Server Tun Mode

CA Backup / Restore Appdata (not running on a schedule), manual only

Community Applications

Dynamix System Buttons

Dynamix System Information

Dynamix System Statistics

Dynamix System Temperature

Fix Common Problems (not on a schedule), manual only

GUI Links

Libvirt Hotplug USB

Nerd Tools - bluez, jqonig, peri, sdparm, expect, tcl, zstd

Resilio

unBALANCE

 

Tunable (enable Direct IO): Auto

 

Thanks for everyone helping in the thread

 

 

 

 

Edited by Addy

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.