Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.

[RichardJones]

Running 6.12.10

Computer: HP 800 SFF with HP 8299 motherboard, I5-7500 CPU at 3.4ghz with 16gb DDR4 ram. I have two 256gb m.2 cache drives in a pool, a 4tb ironwolf for data and a second for parity and a 1tb nvme for Immich.

 

Yesterday whilst transfering movie files to my Media folder Jellyfin (inside a docker container) had a wobbly, stopped working and then the WebUI wouldn't open as it won't connect to the server (all other apps connected are fine). It constantly kicks out the below in the logs. A search identified it could be insufficient storage but my Cache Drive Pool has plenty of space (using 60gb of 256gb)  and I have over 2TB on my storage disk. Let me know if Diagnostics is needed along with the below. I have tried restarting the server which didn't help. Thanks

 

[13:16:11] [FTL] [1] Main: Error while starting server
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
   at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
   at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader()
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteNonQuery()
   at Emby.Server.Implementations.Data.SqliteExtensions.Execute(SqliteConnection sqliteConnection, String commandText)
   at Emby.Server.Implementations.Data.BaseSqliteRepository.Initialize()
   at Emby.Server.Implementations.Data.SqliteItemRepository.Initialize()
   at Emby.Server.Implementations.ApplicationHost.InitializeServices()
   at Jellyfin.Server.Program.StartServer(IServerApplicationPaths appPaths, StartupOptions options, IConfiguration startupConfig)
[13:16:11] [INF] [1] Main: Running query planner optimizations in the database... This might take a while
[13:16:11] [INF] [1] Emby.Server.Implementations.ApplicationHost: Disposing CoreAppHost
[13:16:11] [INF] [1] Emby.Server.Implementations.ApplicationHost: Disposing PluginManager
[13:16:15] [INF] [1] Main: Jellyfin version: 10.9.1
[13:16:15] [INF] [1] Main: Environment Variables: ["[JELLYFIN_CONFIG_DIR, /config]", "[JELLYFIN_WEB_DIR, /usr/share/jellyfin/web]", "[JELLYFIN_LOG_DIR, /config/log]", "[JELLYFIN_DATA_DIR, /config/data]", "[JELLYFIN_CACHE_DIR, /config/cache]"]
[13:16:15] [INF] [1] Main: Arguments: ["/usr/lib/jellyfin/bin/jellyfin.dll", "--ffmpeg=/usr/lib/jellyfin-ffmpeg/ffmpeg"]
[13:16:15] [INF] [1] Main: Operating system: Ubuntu 22.04.4 LTS
[13:16:15] [INF] [1] Main: Architecture: X64
[13:16:15] [INF] [1] Main: 64-Bit Process: True
[13:16:15] [INF] [1] Main: User Interactive: True
[13:16:15] [INF] [1] Main: Processor count: 4
[13:16:15] [INF] [1] Main: Program data path: /config/data
[13:16:15] [INF] [1] Main: Log directory path: /config/log
[13:16:15] [INF] [1] Main: Config directory path: /config
[13:16:15] [INF] [1] Main: Cache path: /config/cache
[13:16:15] [INF] [1] Main: Web resources path: /usr/share/jellyfin/web
[13:16:15] [INF] [1] Main: Application directory: /usr/lib/jellyfin/bin/
[13:16:15] [INF] [1] Emby.Server.Implementations.AppBase.BaseConfigurationManager: Setting cache path: /config/cache
[13:16:19] [INF] [1] Emby.Server.Implementations.ApplicationHost: Loading assemblies
[13:16:19] [INF] [1] Jellyfin.Networking.Manager.NetworkManager: Defined LAN addresses: ["127.0.0.1/8", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]
[13:16:19] [INF] [1] Jellyfin.Networking.Manager.NetworkManager: Defined LAN exclusions: []
[13:16:19] [INF] [1] Jellyfin.Networking.Manager.NetworkManager: Using LAN addresses: ["127.0.0.1/8", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]
[13:16:19] [INF] [1] Jellyfin.Networking.Manager.NetworkManager: Using bind addresses: ["127.0.0.1", "172.20.0.4"]
[13:16:19] [INF] [1] Jellyfin.Networking.Manager.NetworkManager: Remote IP filter is Allowlist
[13:16:19] [INF] [1] Jellyfin.Networking.Manager.NetworkManager: Filter list: []
[13:16:20] [FTL] [1] Main: Error while starting server
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.

cerberus-diagnostics-20240515-1348.zip

Edited May 15 by RichardJones
Added Diagnostics.

[JorgeB]

Please post the diagnostics.
 

[RichardJones]

Sorry added.

[JorgeB]

May 15 09:35:16 Cerberus kernel: BTRFS critical (device nvme1n1p1): unable to find chunk map for logical 9408019707108491264 length 4096
May 15 09:35:16 Cerberus kernel: BTRFS critical (device nvme1n1p1): unable to find chunk map for logical 9408019707108491264 length 16384

 

Pool filesystem issues, you may need to backup and reformat, first change the docker network to ipvlan, reboot, and post new diags after array start.

[RichardJones]

Changed Docker to ipvlan and attached are the diags after array start.

cerberus-diagnostics-20240515-1444.zip

[JorgeB]

Pool filesystem issues are back, try to copy what you can from the pool and then reformat.

[RichardJones]

Ok thanks I have an unnassigned NVME I can try copying to and go from there.

[RichardJones]

Argh - I went into file manager and the Cache folder and copied it to the newly mounted unassigned drive that was located in a Disks folder. This seemed to work with the Cache info copied across. I stopped the array and reformatted the Cache drive. Started the array and the Cache drives were back to 0gb. Now though I cannot see where they are to copy the files back from the unassigned drive to those used for the Cache. If I go back to file manager and click on Cache it shows the unassigned drive. So currently no docker etc working.

[JorgeB]

Not sure I understand, if the new cache is formatted it will be under /mnt/cache

[RichardJones]

Sorry I was being a complete Melon. All sorted and Docker is back up and running. Jellyfin still won't work. Diagnostics attached.

cerberus-diagnostics-20240515-1735.zip

[JorgeB]

Docker image is corrupt, delete and recreate:

 

https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file

Also see below if you have any custom docker networks:

https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks

[RichardJones]

Thanks Jorge - I think all is good! I am logged back in to Jellyfin and setting it up. It's going to take a while to see all the files. Enjoy the beer.  

Edited May 15 by RichardJones