[Solved] Some Containers Work, Others Don't

[brando56894]

Solution: So it turns out FUSE is actually the culprit....and the answer was right here in the LinuxIO Plex support thread. Apparently FUSE freaks out when the cache status for a share is set to cache only and the appdata path for the container is set to /mnt/user/appdata instead of /mnt/cache/appdata, it would be nice if this was included in the FAQ topic of somewhere easily findable instead of having to google "docker shm error" and hunting through tons of random posts. 

 

 

I have about 10 containers setup, and about half of my containers work, while others don't. Even some that used to work, like Plex, are now giving SQlite errors. Deleting said corrupted files and starting the container (or even reinstalling it) result in the same errors.

 

Issues

binhex: sonarr

linuxserver.io: sonarr, radarr, nzbHydra, plex

emby: EmbyServer

diginc: pi-hole

 

Working

google: cadvisor

linuxserver: couchpotato, deluge, htpc-manager, letsencrypt, nextcloud, plexpy

thraxis: transmission-nzbtomedia

 

 

Example of errors from Sonarr/Radarr

17-7-28 19:40:04.9|Info|Bootstrap|Starting Radarr - /opt/radarr/Radarr.exe - Version 0.2.0.778
17-7-28 19:40:05.7|Info|AppFolderInfo|Data directory is being overridden to [/config]
17-7-28 19:40:07.8|Info|MigrationLogger|*** Migrating data source=/config/nzbdrone.db;cache size=-10485760;datetimekind=Utc;journal mode=Wal;pooling=True;version=3 ***
17-7-28 19:40:07.9|Fatal|ConsoleApp|EPIC FAIL!

[v0.2.0.778] NzbDrone.Core.Datastore.CorruptDatabaseException: Database file: /config/nzbdrone.db is corrupt, restore from backup if available. See: https://github.com/Sonarr/Sonarr/wiki/FAQ#i-am-getting-an-error-database-disk-image-is-malformed ---> System.Data.SQLite.SQLiteException: disk I/O error
disk I/O error
  at System.Data.SQLite.SQLite3.Reset (System.Data.SQLite.SQLiteStatement stmt) [0x00083] in <61a20cde294d4a3eb43b9d9f6284613b>:0
  at System.Data.SQLite.SQLite3.Step (System.Data.SQLite.SQLiteStatement stmt) [0x0003c] in <61a20cde294d4a3eb43b9d9f6284613b>:0
  at System.Data.SQLite.SQLiteDataReader.NextResult () [0x0016b] in <61a20cde294d4a3eb43b9d9f6284613b>:0
  at System.Data.SQLite.SQLiteDataReader..ctor (System.Data.SQLite.SQLiteCommand cmd, System.Data.CommandBehavior behave) [0x00090] in <61a20cde294d4a3eb43b9d9f6284613b>:0
  at (wrapper remoting-invoke-with-check) System.Data.SQLite.SQLiteDataReader:.ctor (System.Data.SQLite.SQLiteCommand,System.Data.CommandBehavior)
  at System.Data.SQLite.SQLiteCommand.ExecuteReader (System.Data.CommandBehavior behavior) [0x0000c] in <61a20cde294d4a3eb43b9d9f6284613b>:0
  at System.Data.SQLite.SQLiteCommand.ExecuteDbDataReader (System.Data.CommandBehavior behavior) [0x00000] in <61a20cde294d4a3eb43b9d9f6284613b>:0
  at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader () [0x00000] in <cdc42ce9980a4941a4ca468faca7f261>:0
  at FluentMigrator.Runner.Processors.SQLite.SQLiteProcessor.Exists (System.String template, System.Object[] args) [0x0001f] in C:\Users\Mark\Source\Repos\fluentmigrator\src\FluentMigrator.Runner\Processors\SQLite\SQLiteProcessor.cs:81
  at FluentMigrator.Runner.Processors.SQLite.SQLiteProcessor.TableExists (System.String schemaName, System.String tableName) [0x00000] in C:\Users\Mark\Source\Repos\fluentmigrator\src\FluentMigrator.Runner\Processors\SQLite\SQLiteProcessor.cs:47
  at FluentMigrator.Runner.VersionLoader.get_AlreadyCreatedVersionTable () [0x00000] in C:\Users\Mark\Source\Repos\fluentmigrator\src\FluentMigrator.Runner\VersionLoader.cs:124
  at FluentMigrator.Runner.VersionLoader.LoadVersionInfo () [0x00028] in C:\Users\Mark\Source\Repos\fluentmigrator\src\FluentMigrator.Runner\VersionLoader.cs:160
  at FluentMigrator.Runner.VersionLoader..ctor (FluentMigrator.Runner.IMigrationRunner runner, FluentMigrator.Infrastructure.IAssemblyCollection assemblies, FluentMigrator.IMigrationConventions conventions) [0x00077] in C:\Users\Mark\Source\Repos\fluentmigrator\src\FluentMigrator.Runner\VersionLoader.cs:50
  at FluentMigrator.Runner.MigrationRunner..ctor (FluentMigrator.Infrastructure.IAssemblyCollection assemblies, FluentMigrator.Runner.Initialization.IRunnerContext runnerContext, FluentMigrator.IMigrationProcessor processor) [0x00167] in C:\Users\Mark\Source\Repos\fluentmigrator\src\FluentMigrator.Runner\MigrationRunner.cs:102
  at FluentMigrator.Runner.MigrationRunner..ctor (System.Reflection.Assembly assembly, FluentMigrator.Runner.Initialization.IRunnerContext runnerContext, FluentMigrator.IMigrationProcessor processor) [0x00000] in C:\Users\Mark\Source\Repos\fluentmigrator\src\FluentMigrator.Runner\MigrationRunner.cs:72
  at NzbDrone.Core.Datastore.Migration.Framework.MigrationController.Migrate (System.String connectionString, NzbDrone.Core.Datastore.Migration.Framework.MigrationContext migrationContext) [0x000b5] in C:\projects\radarr-usby1\src\NzbDrone.Core\Datastore\Migration\Framework\MigrationController.cs:58
  at NzbDrone.Core.Datastore.DbFactory.Create (NzbDrone.Core.Datastore.Migration.Framework.MigrationContext migrationContext) [0x00048] in C:\projects\radarr-usby1\src\NzbDrone.Core\Datastore\DbFactory.cs:84
   --- End of inner exception stack trace ---
  at NzbDrone.Core.Datastore.DbFactory.Create (NzbDrone.Core.Datastore.Migration.Framework.MigrationContext migrationContext) [0x00121] in C:\projects\radarr-usby1\src\NzbDrone.Core\Datastore\DbFactory.cs:116
  at NzbDrone.Core.Datastore.DbFactory.Create (NzbDrone.Core.Datastore.Migration.Framework.MigrationType migrationType) [0x00000] in C:\projects\radarr-usby1\src\NzbDrone.Core\Datastore\DbFactory.cs:56
  at NzbDrone.Core.Datastore.DbFactory.RegisterDatabase (NzbDrone.Common.Composition.IContainer container) [0x00000] in C:\projects\radarr-usby1\src\NzbDrone.Core\Datastore\DbFactory.cs:36

 

Sane permissions inside the container

 

root@c05f7f2d2c2a:/config# ls -lh
total 44K
-rw-r--r-- 1 abc users   46 Jul 28 19:40 config.xml
drwxr-xr-x 1 abc users   20 Jul 28 19:40 logs
-rw-r--r-- 1 abc users 1.0K Jul 28 19:40 nzbdrone.db
-rw-r--r-- 1 abc users  32K Jul 28 19:46 nzbdrone.db-shm
-rw-r--r-- 1 abc users    0 Jul 28 19:40 nzbdrone.db-wal
-rw-r--r-- 1 abc users    4 Jul 28 19:47 nzbdrone.pid

 

Sane permissions in unRaid

 

root@Tower:/# ls -lh /mnt/cache/appdata/radarr/
total 44K
-rw-r--r-- 1 nobody users   46 Jul 28 19:40 config.xml
drwxr-xr-x 1 nobody users   20 Jul 28 19:40 logs/
-rw-r--r-- 1 nobody users 1.0K Jul 28 19:40 nzbdrone.db
-rw-r--r-- 1 nobody users  32K Jul 28 19:48 nzbdrone.db-shm
-rw-r--r-- 1 nobody users    0 Jul 28 19:40 nzbdrone.db-wal
-rw-r--r-- 1 nobody users    4 Jul 28 19:48 nzbdrone.pid

 

cache isn't mounted read-only

 

root@Tower:/# mount|grep cache
/dev/sdg1 on /mnt/cache type btrfs (rw,noatime,nodiratime)

Edited July 30, 2017 by brando56894

[Squid]

17 minutes ago, brando56894 said:

/config/nzbdrone.db is corrupt, restore from backup if available. See: https://github.com/Sonarr/Sonarr/wiki/FAQ#i-am-getting-an-error-database-disk-image-is-malformed ---> System.Data.SQLite.SQLiteException: disk I/O error disk I/O error

 

You really should post your diagnostics as it does appear that there is corruption on it.  Once posted, perhaps @johnnie.black can help you.

[brando56894]

Keep on forgetting to use that function since FreeNAS or vanilla Linux doesn't have something that easy haha

 

Here ya go:

 

tower-diagnostics-20170728-2010.zip

[JorgeB]

Just time for a quick look, don't see anything about docker issues in the logs, there are some ATA errors resulting in 2 disable disks, but since the OP didn't mention them I assume the disks were removed on purpose.

 

 

 

 

[brando56894]

3 hours ago, johnnie.black said:

there are some ATA errors resulting in 2 disable disks, but since the OP didn't mention them I assume the disks were removed on purpose.

 

Yea my HBA decided to crap out for some reason when I was removing the 4x 1 TB drives. I have 8 drives that I wanted to use (7x 4 TB, 1x 512 GB SSD) but I only have 6 SATA ports, so I decided to disconnect and create a new config for the drives that only had data on them until I get my new motherboard on Thursday.

 

Edit: I've tried switching to XFS for my cache drive, removing all of my docker containers, setting a share to only use one HDD and bypassing the cache 100%, and even trying three different Sonarr containers and got the same error every time!!

 

Edit 2:

 

It looks to be like it's an issue with the config directory...but it's not. I successfully DD'd an image (as root) from inside a container to the disk without an issue. I also started NzbDrone from within the container (as root) and pointed the config directory at /config and it crashed with the same error, but I pointed it at /root and it started without issue!

 

root@9797b8b550f0:/# mono --debug /opt/NzbDrone/NzbDrone.exe -nobrowser -data=/root
[Info] Bootstrap: Starting Sonarr - /opt/NzbDrone/NzbDrone.exe - Version 2.0.0.4855
[Info] AppFolderInfo: Data directory is being overridden to [/root]
[Info] MigrationLogger: *** Migrating data source=/root/nzbdrone.db;cache size=-10485760;datetimekind=Utc;journal mode=Wal;pooling=True;version=3 ***
[Info] MigrationLogger: *** VersionMigration migrating ***
[Info] MigrationLogger: *** VersionUniqueMigration migrating ***
[Info] MigrationLogger: *** VersionDescriptionMigration migrating ***
[Info] MigrationLogger: *** 1: InitialSetup migrating ***
[Info] InitialSetup: Starting migration to 1
[Info] MigrationLogger: *** 2: remove_tvrage_imdb_unique_constraint migrating ***
[Info] remove_tvrage_imdb_unique_constraint: Starting migration to 2
[Info] MigrationLogger: *** 3: remove_renamed_scene_mapping_columns migrating ***
[Info] remove_renamed_scene_mapping_columns: Starting migration to 3
[Info] MigrationLogger: *** 4: updated_history migrating ***
[Info] updated_history: Starting migration to 4
[Info] MigrationLogger: *** 5: added_eventtype_to_history migrating ***
[Info] added_eventtype_to_history: Starting migration to 5
[Info] MigrationLogger: *** 6: add_index_to_log_time migrating ***

truncated....

 

Edited July 29, 2017 by brando56894

[brando56894]

Even replacing the working SQlite database created in /root with the one in /config causes the same issues, so it's something with SQlite and Docker's transfer protocol that is causing the issues.

 

Here's the errors from binhex-plexpass

 

Jul 29, 2017 20:05:04.728 [0x2b2145d99700] INFO - Plex Media Server v1.7.6.4058-8fa494d15 - ubuntu PC x86_64 - build: linux-ubuntu-x86_64 redhat - GMT -04:00
Jul 29, 2017 20:05:04.728 [0x2b2145d99700] INFO - Linux version: 4.9.30-unRAID (#1 SMP PREEMPT Fri May 26 13:56:36 PDT 2017), language: en-GB
Jul 29, 2017 20:05:04.728 [0x2b2145d99700] INFO - Processor Intel(R) Xeon(R) CPU D-1540 @ 2.00GHz
Jul 29, 2017 20:05:04.728 [0x2b2145d99700] INFO - /opt/plexmediaserver/Plex Media Server
Jul 29, 2017 20:05:04.726 [0x2b213a5856c0] DEBUG - BPQ: [Idle] -> [Starting]
Jul 29, 2017 20:05:04.728 [0x2b213a5856c0] DEBUG - Opening 20 database sessions to library (com.plexapp.plugins.library), SQLite 3.13.0, threadsafe=1
Jul 29, 2017 20:05:04.729 [0x2b213a5856c0] ERROR - SQLITE3:0x10, 5386, os_unix.c:33579: (19) mmap(/config/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db-shm) - No such device
Jul 29, 2017 20:05:04.729 [0x2b213a5856c0] ERROR - SQLITE3:0x10, 5386, disk I/O error
Jul 29, 2017 20:05:04.730 [0x2b213a5856c0] ERROR - Database corruption: sqlite3_statement_backend::prepare: disk I/O error for SQL: PRAGMA cache_size=2000
Jul 29, 2017 20:05:04.730 [0x2b213a5856c0] ERROR - Error: Unable to set up server: sqlite3_statement_backend::prepare: disk I/O error for SQL: PRAGMA cache_size=2000 (N4soci10soci_errorE)

Edited July 30, 2017 by brando56894

[coppit]

I've been dealing with frequent sqlite DB errors in Sonarr. I'm hopeful this was the cause. Thanks for posting the disk I/O errors. That helped me find this solution.