9/17/2019 Update: may have got to the bottom of this. Please try 6.7.3-rc3 available on the next branch.
9/18/2019 Update: 6.7.3-rc4 is available to address Very Slow Array Concurrent Performance.
re:
Trying to get to the bottom of this... First we have not been able to reproduce, which is odd because it implies there may be some kind of hardware/driver dependency with this issue. Nevertheless I want to start a series of tests, which I know will be painful for some since every time DB corruption occurs, you have to go through lengthy rebuild process. That said, we would really appreciate anyone's input during this time.
The idea is that we are only going to change one thing at a time. We can either start with 6.6.7 and start updating stuff until it breaks, or we can start with 6.7.2 and revert stuff until it's fixed. Since my best guess at this point is that the issue is either with Linux kernel, docker, or something we have misconfigured (not one of a hundred other packages we updated), we are going to start with 6.7.2 code base and see if we can make it work.
But actually, the first stab at this is not reverting anything, but rather first updating the Linux kernel to the latest 4.19 patch release which is 4.19.60 (6.7.2 uses kernel 4.19.55). In skimming the kernel change logs, nothing jumps out as a possible fix, however I want to first try the easiest and least impactful change: update to latest 4.19 kernel.
If this does not solve the problem (which I expect it won't), then we have two choices:
1) update to latest Linux stable kernel (5.2.2) - we are using 5.2 kernel in Unraid 6.8-beta and so far no one has reported any sqlite DB corruption, though the sample set is pretty small. The downside with this is, not all out-of-tree drivers yet build with 5.2 kernel and so some functionality would be lost.
2) downgrade docker from 18.09.06 (version in 6.7.2) to 18.06.03-ce (version in 6.6.7).
[BTW the latest Docker release 19.03.00 was just published today - people gripe about our release numbers, try making sense of Docker release numbers haha]
If neither of those steps succeed then ... well let's hope one of them does succeed.
To get started, first make a backup of your flash via Main/Flash/Flash Backup, and then switch to the 'next' branch via Tools/Upgrade OS page. There you should see version 6.7.3-rc1
As soon as a couple people report corruption I'll publish an -rc2, probably with reverted Docker.
Edited by limetech
- 5
Recommended Comments