brando56894 Posted July 28, 2017 Share Posted July 28, 2017 (edited) Solution: So it turns out FUSE is actually the culprit....and the answer was right here in the LinuxIO Plex support thread. Apparently FUSE freaks out when the cache status for a share is set to cache only and the appdata path for the container is set to /mnt/user/appdata instead of /mnt/cache/appdata, it would be nice if this was included in the FAQ topic of somewhere easily findable instead of having to google "docker shm error" and hunting through tons of random posts. I have about 10 containers setup, and about half of my containers work, while others don't. Even some that used to work, like Plex, are now giving SQlite errors. Deleting said corrupted files and starting the container (or even reinstalling it) result in the same errors. Issues binhex: sonarr linuxserver.io: sonarr, radarr, nzbHydra, plex emby: EmbyServer diginc: pi-hole Working google: cadvisor linuxserver: couchpotato, deluge, htpc-manager, letsencrypt, nextcloud, plexpy thraxis: transmission-nzbtomedia Example of errors from Sonarr/Radarr 17-7-28 19:40:04.9|Info|Bootstrap|Starting Radarr - /opt/radarr/Radarr.exe - Version 0.2.0.778 17-7-28 19:40:05.7|Info|AppFolderInfo|Data directory is being overridden to [/config] 17-7-28 19:40:07.8|Info|MigrationLogger|*** Migrating data source=/config/nzbdrone.db;cache size=-10485760;datetimekind=Utc;journal mode=Wal;pooling=True;version=3 *** 17-7-28 19:40:07.9|Fatal|ConsoleApp|EPIC FAIL! [v0.2.0.778] NzbDrone.Core.Datastore.CorruptDatabaseException: Database file: /config/nzbdrone.db is corrupt, restore from backup if available. See: https://github.com/Sonarr/Sonarr/wiki/FAQ#i-am-getting-an-error-database-disk-image-is-malformed ---> System.Data.SQLite.SQLiteException: disk I/O error disk I/O error at System.Data.SQLite.SQLite3.Reset (System.Data.SQLite.SQLiteStatement stmt) [0x00083] in <61a20cde294d4a3eb43b9d9f6284613b>:0 at System.Data.SQLite.SQLite3.Step (System.Data.SQLite.SQLiteStatement stmt) [0x0003c] in <61a20cde294d4a3eb43b9d9f6284613b>:0 at System.Data.SQLite.SQLiteDataReader.NextResult () [0x0016b] in <61a20cde294d4a3eb43b9d9f6284613b>:0 at System.Data.SQLite.SQLiteDataReader..ctor (System.Data.SQLite.SQLiteCommand cmd, System.Data.CommandBehavior behave) [0x00090] in <61a20cde294d4a3eb43b9d9f6284613b>:0 at (wrapper remoting-invoke-with-check) System.Data.SQLite.SQLiteDataReader:.ctor (System.Data.SQLite.SQLiteCommand,System.Data.CommandBehavior) at System.Data.SQLite.SQLiteCommand.ExecuteReader (System.Data.CommandBehavior behavior) [0x0000c] in <61a20cde294d4a3eb43b9d9f6284613b>:0 at System.Data.SQLite.SQLiteCommand.ExecuteDbDataReader (System.Data.CommandBehavior behavior) [0x00000] in <61a20cde294d4a3eb43b9d9f6284613b>:0 at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader () [0x00000] in <cdc42ce9980a4941a4ca468faca7f261>:0 at FluentMigrator.Runner.Processors.SQLite.SQLiteProcessor.Exists (System.String template, System.Object[] args) [0x0001f] in C:\Users\Mark\Source\Repos\fluentmigrator\src\FluentMigrator.Runner\Processors\SQLite\SQLiteProcessor.cs:81 at FluentMigrator.Runner.Processors.SQLite.SQLiteProcessor.TableExists (System.String schemaName, System.String tableName) [0x00000] in C:\Users\Mark\Source\Repos\fluentmigrator\src\FluentMigrator.Runner\Processors\SQLite\SQLiteProcessor.cs:47 at FluentMigrator.Runner.VersionLoader.get_AlreadyCreatedVersionTable () [0x00000] in C:\Users\Mark\Source\Repos\fluentmigrator\src\FluentMigrator.Runner\VersionLoader.cs:124 at FluentMigrator.Runner.VersionLoader.LoadVersionInfo () [0x00028] in C:\Users\Mark\Source\Repos\fluentmigrator\src\FluentMigrator.Runner\VersionLoader.cs:160 at FluentMigrator.Runner.VersionLoader..ctor (FluentMigrator.Runner.IMigrationRunner runner, FluentMigrator.Infrastructure.IAssemblyCollection assemblies, FluentMigrator.IMigrationConventions conventions) [0x00077] in C:\Users\Mark\Source\Repos\fluentmigrator\src\FluentMigrator.Runner\VersionLoader.cs:50 at FluentMigrator.Runner.MigrationRunner..ctor (FluentMigrator.Infrastructure.IAssemblyCollection assemblies, FluentMigrator.Runner.Initialization.IRunnerContext runnerContext, FluentMigrator.IMigrationProcessor processor) [0x00167] in C:\Users\Mark\Source\Repos\fluentmigrator\src\FluentMigrator.Runner\MigrationRunner.cs:102 at FluentMigrator.Runner.MigrationRunner..ctor (System.Reflection.Assembly assembly, FluentMigrator.Runner.Initialization.IRunnerContext runnerContext, FluentMigrator.IMigrationProcessor processor) [0x00000] in C:\Users\Mark\Source\Repos\fluentmigrator\src\FluentMigrator.Runner\MigrationRunner.cs:72 at NzbDrone.Core.Datastore.Migration.Framework.MigrationController.Migrate (System.String connectionString, NzbDrone.Core.Datastore.Migration.Framework.MigrationContext migrationContext) [0x000b5] in C:\projects\radarr-usby1\src\NzbDrone.Core\Datastore\Migration\Framework\MigrationController.cs:58 at NzbDrone.Core.Datastore.DbFactory.Create (NzbDrone.Core.Datastore.Migration.Framework.MigrationContext migrationContext) [0x00048] in C:\projects\radarr-usby1\src\NzbDrone.Core\Datastore\DbFactory.cs:84 --- End of inner exception stack trace --- at NzbDrone.Core.Datastore.DbFactory.Create (NzbDrone.Core.Datastore.Migration.Framework.MigrationContext migrationContext) [0x00121] in C:\projects\radarr-usby1\src\NzbDrone.Core\Datastore\DbFactory.cs:116 at NzbDrone.Core.Datastore.DbFactory.Create (NzbDrone.Core.Datastore.Migration.Framework.MigrationType migrationType) [0x00000] in C:\projects\radarr-usby1\src\NzbDrone.Core\Datastore\DbFactory.cs:56 at NzbDrone.Core.Datastore.DbFactory.RegisterDatabase (NzbDrone.Common.Composition.IContainer container) [0x00000] in C:\projects\radarr-usby1\src\NzbDrone.Core\Datastore\DbFactory.cs:36 Sane permissions inside the container root@c05f7f2d2c2a:/config# ls -lh total 44K -rw-r--r-- 1 abc users 46 Jul 28 19:40 config.xml drwxr-xr-x 1 abc users 20 Jul 28 19:40 logs -rw-r--r-- 1 abc users 1.0K Jul 28 19:40 nzbdrone.db -rw-r--r-- 1 abc users 32K Jul 28 19:46 nzbdrone.db-shm -rw-r--r-- 1 abc users 0 Jul 28 19:40 nzbdrone.db-wal -rw-r--r-- 1 abc users 4 Jul 28 19:47 nzbdrone.pid Sane permissions in unRaid root@Tower:/# ls -lh /mnt/cache/appdata/radarr/ total 44K -rw-r--r-- 1 nobody users 46 Jul 28 19:40 config.xml drwxr-xr-x 1 nobody users 20 Jul 28 19:40 logs/ -rw-r--r-- 1 nobody users 1.0K Jul 28 19:40 nzbdrone.db -rw-r--r-- 1 nobody users 32K Jul 28 19:48 nzbdrone.db-shm -rw-r--r-- 1 nobody users 0 Jul 28 19:40 nzbdrone.db-wal -rw-r--r-- 1 nobody users 4 Jul 28 19:48 nzbdrone.pid cache isn't mounted read-only root@Tower:/# mount|grep cache /dev/sdg1 on /mnt/cache type btrfs (rw,noatime,nodiratime) Edited July 30, 2017 by brando56894 Quote Link to comment
Squid Posted July 29, 2017 Share Posted July 29, 2017 17 minutes ago, brando56894 said: /config/nzbdrone.db is corrupt, restore from backup if available. See: https://github.com/Sonarr/Sonarr/wiki/FAQ#i-am-getting-an-error-database-disk-image-is-malformed ---> System.Data.SQLite.SQLiteException: disk I/O error disk I/O error You really should post your diagnostics as it does appear that there is corruption on it. Once posted, perhaps @johnnie.black can help you. Quote Link to comment
brando56894 Posted July 29, 2017 Author Share Posted July 29, 2017 Keep on forgetting to use that function since FreeNAS or vanilla Linux doesn't have something that easy haha Here ya go: tower-diagnostics-20170728-2010.zip Quote Link to comment
JorgeB Posted July 29, 2017 Share Posted July 29, 2017 Just time for a quick look, don't see anything about docker issues in the logs, there are some ATA errors resulting in 2 disable disks, but since the OP didn't mention them I assume the disks were removed on purpose. Quote Link to comment
brando56894 Posted July 29, 2017 Author Share Posted July 29, 2017 (edited) 3 hours ago, johnnie.black said: there are some ATA errors resulting in 2 disable disks, but since the OP didn't mention them I assume the disks were removed on purpose. Yea my HBA decided to crap out for some reason when I was removing the 4x 1 TB drives. I have 8 drives that I wanted to use (7x 4 TB, 1x 512 GB SSD) but I only have 6 SATA ports, so I decided to disconnect and create a new config for the drives that only had data on them until I get my new motherboard on Thursday. Edit: I've tried switching to XFS for my cache drive, removing all of my docker containers, setting a share to only use one HDD and bypassing the cache 100%, and even trying three different Sonarr containers and got the same error every time!! Edit 2: It looks to be like it's an issue with the config directory...but it's not. I successfully DD'd an image (as root) from inside a container to the disk without an issue. I also started NzbDrone from within the container (as root) and pointed the config directory at /config and it crashed with the same error, but I pointed it at /root and it started without issue! root@9797b8b550f0:/# mono --debug /opt/NzbDrone/NzbDrone.exe -nobrowser -data=/root [Info] Bootstrap: Starting Sonarr - /opt/NzbDrone/NzbDrone.exe - Version 2.0.0.4855 [Info] AppFolderInfo: Data directory is being overridden to [/root] [Info] MigrationLogger: *** Migrating data source=/root/nzbdrone.db;cache size=-10485760;datetimekind=Utc;journal mode=Wal;pooling=True;version=3 *** [Info] MigrationLogger: *** VersionMigration migrating *** [Info] MigrationLogger: *** VersionUniqueMigration migrating *** [Info] MigrationLogger: *** VersionDescriptionMigration migrating *** [Info] MigrationLogger: *** 1: InitialSetup migrating *** [Info] InitialSetup: Starting migration to 1 [Info] MigrationLogger: *** 2: remove_tvrage_imdb_unique_constraint migrating *** [Info] remove_tvrage_imdb_unique_constraint: Starting migration to 2 [Info] MigrationLogger: *** 3: remove_renamed_scene_mapping_columns migrating *** [Info] remove_renamed_scene_mapping_columns: Starting migration to 3 [Info] MigrationLogger: *** 4: updated_history migrating *** [Info] updated_history: Starting migration to 4 [Info] MigrationLogger: *** 5: added_eventtype_to_history migrating *** [Info] added_eventtype_to_history: Starting migration to 5 [Info] MigrationLogger: *** 6: add_index_to_log_time migrating *** truncated.... Edited July 29, 2017 by brando56894 Quote Link to comment
brando56894 Posted July 29, 2017 Author Share Posted July 29, 2017 (edited) Even replacing the working SQlite database created in /root with the one in /config causes the same issues, so it's something with SQlite and Docker's transfer protocol that is causing the issues. Here's the errors from binhex-plexpass Jul 29, 2017 20:05:04.728 [0x2b2145d99700] INFO - Plex Media Server v1.7.6.4058-8fa494d15 - ubuntu PC x86_64 - build: linux-ubuntu-x86_64 redhat - GMT -04:00 Jul 29, 2017 20:05:04.728 [0x2b2145d99700] INFO - Linux version: 4.9.30-unRAID (#1 SMP PREEMPT Fri May 26 13:56:36 PDT 2017), language: en-GB Jul 29, 2017 20:05:04.728 [0x2b2145d99700] INFO - Processor Intel(R) Xeon(R) CPU D-1540 @ 2.00GHz Jul 29, 2017 20:05:04.728 [0x2b2145d99700] INFO - /opt/plexmediaserver/Plex Media Server Jul 29, 2017 20:05:04.726 [0x2b213a5856c0] DEBUG - BPQ: [Idle] -> [Starting] Jul 29, 2017 20:05:04.728 [0x2b213a5856c0] DEBUG - Opening 20 database sessions to library (com.plexapp.plugins.library), SQLite 3.13.0, threadsafe=1 Jul 29, 2017 20:05:04.729 [0x2b213a5856c0] ERROR - SQLITE3:0x10, 5386, os_unix.c:33579: (19) mmap(/config/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db-shm) - No such device Jul 29, 2017 20:05:04.729 [0x2b213a5856c0] ERROR - SQLITE3:0x10, 5386, disk I/O error Jul 29, 2017 20:05:04.730 [0x2b213a5856c0] ERROR - Database corruption: sqlite3_statement_backend::prepare: disk I/O error for SQL: PRAGMA cache_size=2000 Jul 29, 2017 20:05:04.730 [0x2b213a5856c0] ERROR - Error: Unable to set up server: sqlite3_statement_backend::prepare: disk I/O error for SQL: PRAGMA cache_size=2000 (N4soci10soci_errorE) Edited July 30, 2017 by brando56894 Quote Link to comment
coppit Posted April 11 Share Posted April 11 I've been dealing with frequent sqlite DB errors in Sonarr. I'm hopeful this was the cause. Thanks for posting the disk I/O errors. That helped me find this solution. Quote Link to comment
permster Posted May 1 Share Posted May 1 I also started having issues just recently with Sonarr database corruption. I can restore from backup and all is fine but then it corrupts again a day or two later. Just this morning I saw all the disk I/O errors in the Sonarr events. My appdata share is set to Primary > Cache and Secondary > Array with mover action set to Array > Cache so I'm not set to cache only but I'm still experiencing the same issue. No other containers have this issue that I'm aware of. Are there any other ways to troubleshoot this issue? I see nothing in syslog when these issues are occurring in the Sonarr container. Unraid - 6.12.9 Sonarr - 4.0.4.1491 Database - Sqlite 3.44.2 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.