ZECHS

For the docker folder thing, I just thought it would be easier to backup and manage as folders. I have no particular preference for it, hopefully it won't be too hard to switch back to image but I will definitely do that if it will be easier to catch errors in the future. As for the shares minimum free stuff, I am still setting this stuff up and am learning about a lot of this as I go. My previous server was a lot simpler with basically just the Servarr stack and plex). I will go over these and set them to something sensible to see if that helps. And the share data stuff. After you told me about the diagnostics zip I started looking through and saw that. When I originally setup the server I made all shares use cache and then my basic pool. Found out quickly this was a no-no for appdata and system and switched them both to just live on my cache nvme until I can pick up another stick. After I switched the storage, I triggered the mover and assumed if everything was working, my work was done. Anyways, I have already gone through and stopped docker service, rsynced the files from /mnt/basic back to /mnt/cache, triple checked no files were left behind, unmounted them from /mnt/basic, and "zfs destroy" on the leftover empty datasets in the basic pool. At the moment my databases are working so gunna give it another 10 mins and if all is working well, I will start looking at/implementing your other points. I'm still getting some of those shares with files in the wrong pool messages but that is a separate issue and I'll be solving that next. Thank you for your time and patience

Sorry about that, here it is magi-diagnostics-20250407-1556.zip

I am working on setting up a new UNRAID server as my current setup is running on a very underpowered machine. This was all going well for the past 3 weeks as I have been configuring things and getting services up and running. This morning I woke up to seeing a lot of my services like Sonarr throwing errors saying they cannot open the database. This is happening on many services all at the same time. Everything with a database connection is unable to open. As i have been looking into the logs of the containers that are failing i also keep getting errors that say i am out of diskspace even though i have 1.5tb free on my appdata/system drive and 2 separate pools with 10tb and 34tb free. [Error] SonarrErrorPipeline: [GET /api/v3/queue] [v4.0.14.2939] code = CantOpen (14), message = System.Data.SQLite.SQLiteException (0x800007FF): unable to open database file unable to open database file at System.Data.SQLite.SQLite3.Prepare(SQLiteConnection cnn, String strSql, SQLiteStatement previous, UInt32 timeoutMS, String& strRemain) at System.Data.SQLite.SQLiteCommand.BuildNextCommand() at System.Data.SQLite.SQLiteDataReader.NextResult() at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave) at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior) at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior) at System.Data.SQLite.SQLiteConnection.Open() at NzbDrone.Core.Datastore.DbFactory.<>c__DisplayClass9_0.<Create>b__0() in ./Sonarr.Core/Datastore/DbFactory.cs:line 105 at NzbDrone.Core.Datastore.SqlMapperExtensions.Query[T](IDatabase db, String sql, Object param) in ./Sonarr.Core/Datastore/Extensions/SqlMapperExtensions.cs:line 12 at NzbDrone.Core.Datastore.BasicRepository`1.Query(SqlBuilder builder) in ./Sonarr.Core/Datastore/BasicRepository.cs:line 72 at NzbDrone.Core.Download.Pending.PendingReleaseService.GetPendingQueue() in ./Sonarr.Core/Download/Pending/PendingReleaseService.cs:line 184 at Sonarr.Api.V3.Queue.QueueController.GetQueue(PagingSpec`1 pagingSpec, HashSet`1 seriesIds, Nullable`1 protocol, HashSet`1 languages, HashSet`1 quality, HashSet`1 status, Boolean includeUnknownSeriesItems) at Sonarr.Api.V3.Queue.QueueController.<>c__DisplayClass13_0.<GetQueue>b__0(PagingSpec`1 spec) in ./Sonarr.Api.V3/Queue/QueueController.cs:line 168 at Sonarr.Http.Extensions.RequestExtensions.ApplyToPage[TResource,TModel](PagingSpec`1 pagingSpec, Func`2 function, Converter`2 mapper) in ./Sonarr.Http/Extensions/RequestExtensions.cs:line 67 at Sonarr.Api.V3.Queue.QueueController.GetQueue(PagingRequestResource paging, Boolean includeUnknownSeriesItems, Boolean includeSeries, Boolean includeEpisode, Int32[] seriesIds, Nullable`1 protocol, Int32[] languages, Int32[] quality, QueueStatus[] status) at lambda_method110(Closure , Object , Object[] ) at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync() at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync() [Error] SonarrErrorPipeline: [GET /api/v3/queue] [v4.0.14.2939] code = CantOpen (14), message = System.Data.SQLite.SQLiteException (0x800007FF): unable to open database file unable to open database file at System.Data.SQLite.SQLite3.Prepare(SQLiteConnection cnn, String strSql, SQLiteStatement previous, UInt32 timeoutMS, String& strRemain) at System.Data.SQLite.SQLiteCommand.BuildNextCommand() at System.Data.SQLite.SQLiteDataReader.NextResult() at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave) at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior) at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior) at System.Data.SQLite.SQLiteConnection.Open() at NzbDrone.Core.Datastore.DbFactory.<>c__DisplayClass9_0.<Create>b__0() in ./Sonarr.Core/Datastore/DbFactory.cs:line 105 at NzbDrone.Core.Datastore.SqlMapperExtensions.Query[T](IDatabase db, String sql, Object param) in ./Sonarr.Core/Datastore/Extensions/SqlMapperExtensions.cs:line 12 at NzbDrone.Core.Datastore.BasicRepository`1.Query(SqlBuilder builder) in ./Sonarr.Core/Datastore/BasicRepository.cs:line 72 at NzbDrone.Core.Download.Pending.PendingReleaseService.GetPendingQueue() in ./Sonarr.Core/Download/Pending/PendingReleaseService.cs:line 184 at Sonarr.Api.V3.Queue.QueueController.GetQueue(PagingSpec`1 pagingSpec, HashSet`1 seriesIds, Nullable`1 protocol, HashSet`1 languages, HashSet`1 quality, HashSet`1 status, Boolean includeUnknownSeriesItems) at Sonarr.Api.V3.Queue.QueueController.<>c__DisplayClass13_0.<GetQueue>b__0(PagingSpec`1 spec) in ./Sonarr.Api.V3/Queue/QueueController.cs:line 168 at Sonarr.Http.Extensions.RequestExtensions.ApplyToPage[TResource,TModel](PagingSpec`1 pagingSpec, Func`2 function, Converter`2 mapper) in ./Sonarr.Http/Extensions/RequestExtensions.cs:line 67 at Sonarr.Api.V3.Queue.QueueController.GetQueue(PagingRequestResource paging, Boolean includeUnknownSeriesItems, Boolean includeSeries, Boolean includeEpisode, Int32[] seriesIds, Nullable`1 protocol, Int32[] languages, Int32[] quality, QueueStatus[] status) at lambda_method110(Closure , Object , Object[] ) at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync() at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync() --- End of stack trace from previous location --- at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync() --- End of stack trace from previous location --- at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger) at Sonarr.Http.Middleware.BufferingMiddleware.InvokeAsync(HttpContext context) in ./Sonarr.Http/Middleware/BufferingMiddleware.cs:line 28 at Sonarr.Http.Middleware.IfModifiedMiddleware.InvokeAsync(HttpContext context) in ./Sonarr.Http/Middleware/IfModifiedMiddleware.cs:line 41 at Sonarr.Http.Middleware.CacheHeaderMiddleware.InvokeAsync(HttpContext context) in ./Sonarr.Http/Middleware/CacheHeaderMiddleware.cs:line 33 at Sonarr.Http.Middleware.StartingUpMiddleware.InvokeAsync(HttpContext context) in ./Sonarr.Http/Middleware/StartingUpMiddleware.cs:line 38 at Sonarr.Http.Middleware.UrlBaseMiddleware.InvokeAsync(HttpContext context) in ./Sonarr.Http/Middleware/UrlBaseMiddleware.cs:line 29 at Sonarr.Http.Middleware.VersionMiddleware.InvokeAsync(HttpContext context) in ./Sonarr.Http/Middleware/VersionMiddleware.cs:line 29 at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context) at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult) at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task) [Error] TaskExtensions: Task Error [v4.0.14.2939] code = CantOpen (14), message = System.Data.SQLite.SQLiteException (0x800007FF): unable to open database file unable to open database file at System.Data.SQLite.SQLite3.Prepare(SQLiteConnection cnn, String strSql, SQLiteStatement previous, UInt32 timeoutMS, String& strRemain) at System.Data.SQLite.SQLiteCommand.BuildNextCommand() at System.Data.SQLite.SQLiteDataReader.NextResult() at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave) at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior) at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior) at System.Data.SQLite.SQLiteConnection.Open() at NzbDrone.Core.Datastore.DbFactory.<>c__DisplayClass9_0.<Create>b__0() in ./Sonarr.Core/Datastore/DbFactory.cs:line 105 at NzbDrone.Core.Datastore.BasicRepository`1.Insert(TModel model) in ./Sonarr.Core/Datastore/BasicRepository.cs:line 151 at NzbDrone.Core.Messaging.Commands.CommandQueueManager.Push[TCommand](TCommand command, CommandPriority priority, CommandTrigger trigger) in ./Sonarr.Core/Messaging/Commands/CommandQueueManager.cs:line 140 at NzbDrone.Core.Jobs.Scheduler.ExecuteCommands() in ./Sonarr.Core/Jobs/Scheduler.cs:line 42 at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) 2025-04-07 18:54:47.486 UTC [1] FATAL: could not create lock file "postmaster.pid": No space left on device 2025-04-07 18:55:47.927 UTC [1] FATAL: could not create lock file "postmaster.pid": No space left on device 2025-04-07 18:56:48.420 UTC [1] FATAL: could not create lock file "postmaster.pid": No space left on device PostgreSQL Database directory appears to contain a database; Skipping initialization PostgreSQL Database directory appears to contain a database; Skipping initialization PostgreSQL Database directory appears to contain a database; Skipping initialization My only other thought was to see if the docker image size had to be increased. I thought this setting used to be in Settings -> Docker but I don't see any option there for it now. A little info on my setup: - All databases stored in appdata, which lives on an SSD pool of only that drive. - All my configs were working days ago and are based off of my other machine which has been running without issue for over a year. - All of my docker containers are done with docker compose - Effected services include; Sonarr, Radarr, Lidarr, Docmost, Nextcloud, Authentik, Gitea, Tautulli, Readarr, essentially anything with a database - I am using the latest version of UNRAID Some things I have tried so far: - Restarting docker service - Restarting UNRAID - Removing all docker containers and networks and redeploying. (this actually got things working for a couple mins, able to load all my data up and everything, but it went back to being unable to open the database - Reviewing configs to see if anything changed (they didn't) At this point I'm not sure how to proceed and any advice would be appreciated.