[Support] Paperless-ng Docker

[Ford Prefect]

...should be OK for the REDIS container...did you check the paperless-ng logs upon startup, whether it can connect to REDIS?

There needs to be the Variable "PAPERLESS_REDIS" setup correctly.... use Protocol, IP and Port, like "redis://<IP-of-your-redis-container>:6379"

 

This is how it looks on my site:

 

[Flight777]

Thanks!

Found it finally, made a typo in the REDIS IP that I overlooked multiple times... 🙈

192.68.xx.xx instead of 192.168.xx.xx

[WillyB]

Just installed this on a brand new UnRAID to replace Evernote! Thanks for the community app setup! 

My mail import did not work so I erased the Mail account in settings and re-added it. 

Since doing that the task fails with: 

 

ERROR Failed [enemy-ceiling-delta-wyoming] - [Errno -2] Name or service not known : Traceback (most recent call last):

File "/usr/local/lib/python3.7/site-packages/django_q/cluster.py", line 436, in worker
res = f(*task["args"], **task["kwargs"])

 

Any idea what's wrong ? Everything else is working. 

[T0a]

4 hours ago, WillyB said:

Just installed this on a brand new UnRAID to replace Evernote! Thanks for the community app setup! 

My mail import did not work so I erased the Mail account in settings and re-added it. 

Since doing that the task fails with: 

 

ERROR Failed [enemy-ceiling-delta-wyoming] - [Errno -2] Name or service not known : Traceback (most recent call last):

File "/usr/local/lib/python3.7/site-packages/django_q/cluster.py", line 436, in worker
res = f(*task["args"], **task["kwargs"])

 

Any idea what's wrong ? Everything else is working. 

 

Have you stripped away parts from the stacktrace? Is this stacktrace from the web UI? Does the docker container log contain more information? For me the error "name or service not known" looks like a typically docker container resolution issue. I've seen such errors when you register or link a service that resides outside the container and the application tries to find it inside container.

 

How does your Redis environment configuration look like? Any chance you linked something else with a symbolic name that is not known to the container? I also wonder why the container name is "enemy-ceiling-delta-wyoming". The paperless container name is fixed to paperless-ng AFAIK or am I wrong?

[WillyB]

I re-installed paperless-ng and redis. 

I no longer get the Traceback error and it imports emails just fine now... 

As for the names, I am not sure, all the completed task show a similar name formula instead of a date/time: 

 

 

[daemian]

EDIT: I figured it out. It was exactly what it sounded like in the error... the file (.pdf) was missing. What I forgot is that some time ago I had an issue with a drive, and I messed up the replacement process. I ended up losing a number of files, including some from the paperless media directory. I assume the migration was failing because the file was referenced in the paperless DB but not there in the folder. I just put a "fummy" pdf in the folder with the appropriate name and started it again. It would then identify a different missing file. I repeated that process until I got all of the missing files replaced, and then the migration ran successfully! Thanks again for you work on this Docker.

 

---Leaving this for posterity, though I doubt anyone else will run into what I did---

 

Thanks for all of your hard work on this docker T0a.

 

I am trying to migrated from an unraid based paperless container I have had for years (thepaperlessproject/paperless) and running into an issue. I may just be doing it entirely wrong....

 

I installed theredis container and then I made copies of my original papaerless appdata and user shares (with media, consume, & export dirs). When deploying the paperless-ng docker I pointed the appdata, consume, export, and media directories to those copies I made. When I start the docker I get this, and then the container stops:

 

Paperless-ng docker container starting...
Mapping UID and GID for paperless:paperless to 99:100
creating directory /tmp/paperless
Apply database migrations...
Operations to perform:
Apply all migrations: admin, auth, authtoken, contenttypes, django_q, documents, paperless_mail, sessions
Running migrations:
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 330, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 371, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 85, in wrapped
res = handle_func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/migrate.py", line 245, in handle
fake_initial=fake_initial,
File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
state = migration.apply(state, schema_editor)
File "/usr/local/lib/python3.7/site-packages/django/db/migrations/migration.py", line 124, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/usr/local/lib/python3.7/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
self.code(from_state.apps, schema_editor)
File "/usr/src/paperless/src/documents/migrations/1003_mime_types.py", line 34, in add_mime_types
f = open(source_path(d), "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/usr/src/paperless/src/../media/documents/originals/0001216.pdf'
Applying documents.1003_mime_types...

 

A snapshot of my docker settings is attached, and here are the directories:

 

[email protected]:~# tree -d /mnt/user/paperless-ng/
/mnt/user/paperless-ng/
├── consume
├── export
├── media
│   └── documents
│       ├── originals
│       └── thumbnails
└── temp

7 directories
[email protected]:~# tree -d /mnt/disks/ssd1/appdata/paperless-ng/
/mnt/disks/ssd1/appdata/paperless-ng/
└── data
    └── index

2 directories
[email protected]:~#

 

 

Any idea what I am doing wrong?

 

Thanks in advanced!

 

 

Edited February 2, 2021 by daemian

[Abe677]

This is a Paperless-ng use question. Have it running on Unraid. If this isn't the right place to ask feel free to delete my post.

 

I know once my document is consumed I can use the web UI to tag my document. I've looked at the suggested work flow. What I'm not clear about are the file name requirements for the scanned documents? There's a brief note about Paperless-ng using the file name to determine things about the document before it does an OCR and other processing, but I haven't stumbled on any suggested file naming for the scanned documents that are going to be pushed to the consume folder. Is there any magic there or does it really not matter? (Maybe the answer is different depending on whether I'm scanning and shredding vs archiving?)

[daemian]

I am not 100% sure about this since I don't really use it, and have only been using paperless-ng for a few days now, but the original version of paperless had a defined format for interpreting date, correspondent, title, tags, etc. from the filename. My guess is that paperless-ng uses this same logic.

See here: https://paperless.readthedocs.io/en/latest/guesswork.html

 

If you get your tags and corespondent text matching setup well you should not have to bother. It will find the info in the OCR text from the coument and set those fields along with the data automatically for you. Of course that doesn't always work - but its pretty good most of the time.

[Michelangelo]

I'm completly new to the wonderfull world of unraid and dockers, so this might be a very noob question. Its just something i can't wrap my head around. 

 

I had to install REDIS, easy as just adding it via the community apps. But there is mapping to a volume, so where does it store its database? Shouldn't I be mapping a folder so that paperless-ng has enough room to store its data? 

[T0a]

On 1/29/2021 at 3:29 PM, daemian said:

EDIT: I figured it out. It was exactly what it sounded like in the error... the file (.pdf) was missing.

Glad you got it working.

 

On 1/29/2021 at 3:29 PM, daemian said:

Thanks for all of your hard work on this docker T0a.

 

No problem, you are welcome. Creating this template was no effort for me tbh. Credits go to the new dev Jonas Winkler for improving paperless.

 

On 2/4/2021 at 8:34 PM, Abe677 said:

This is a Paperless-ng use question. Have it running on Unraid. If this isn't the right place to ask feel free to delete my post.

 

I know once my document is consumed I can use the web UI to tag my document. I've looked at the suggested work flow. What I'm not clear about are the file name requirements for the scanned documents? There's a brief note about Paperless-ng using the file name to determine things about the document before it does an OCR and other processing, but I haven't stumbled on any suggested file naming for the scanned documents that are going to be pushed to the consume folder. Is there any magic there or does it really not matter? (Maybe the answer is different depending on whether I'm scanning and shredding vs archiving?)

On 2/4/2021 at 8:49 PM, daemian said:

I am not 100% sure about this since I don't really use it, and have only been using paperless-ng for a few days now, but the original version of paperless had a defined format for interpreting date, correspondent, title, tags, etc. from the filename. My guess is that paperless-ng uses this same logic.

See here: https://paperless.readthedocs.io/en/latest/guesswork.html

 

If you get your tags and corespondent text matching setup well you should not have to bother. It will find the info in the OCR text from the coument and set those fields along with the data automatically for you. Of course that doesn't always work - but its pretty good most of the time.

 

There is no requirement for the document name.  The guesswork feature you are referring to was removed in version 0.9.9.

 

35 minutes ago, Michelangelo said:

I'm completly new to the wonderfull world of unraid and dockers, so this might be a very noob question. Its just something i can't wrap my head around. 

 

I had to install REDIS, easy as just adding it via the community apps. But there is mapping to a volume, so where does it store its database? Shouldn't I be mapping a folder so that paperless-ng has enough room to store its data? 

 

The Redis database is just used for checking if a document is finished processing. Make sure you replace the IP in the template with the machine running the Redis database. The documents are stored in the Media folder. Make sure that this host mount points to a share on your server with enough space for the documents. The documents metadata get persisted in a SQLite database that is stored in the Data directory. It is also possible to use a full-blown SQL database such as PostgreSQL rather then the file based database. The steps to migrate from the default SQLite database to PostgreSQL are outlined in the first page of this thread. Though, documents will still be stored as files on the mapped Media share. Only metadata is persisted in the database.

Edited February 6, 2021 by T0a

[Michelangelo]

14 minutes ago, T0a said:

The Redis database is just used for checking if a document is finished processing. Make sure you replace the IP in the template with the machine running the Redis database. The documents are stored in the Media folder. Make sure that this host mount points to a share on your server with enough space for the documents. The documents metadata get persisted in a SQLite database that is stored in the Data directory. It is also possible to use a full-blown SQL database such as PostgreSQL rather then the file based database. The steps to migrate from the default SQLite database to PostgreSQL are outlined in the first page of this thread. Though, documents will still be stored as files on the mapped Media share. Only metadata is persisted in the database.

 

Ah that makes sense! Thanks for clarifying that! What would be the reason to use PostGreSQL if you are not using the database to store data itself? I assume SQlite is more than capable of doing the job, right? 

[T0a]

7 minutes ago, Michelangelo said:

 

Ah that makes sense! Thanks for clarifying that! What would be the reason to use PostGreSQL if you are not using the database to store data itself? I assume SQlite is more than capable of doing the job, right? 

 

I think concurrent access to a file based database such as SQLite is kind of a problem sometimes and results in issues like "database is locked" messages. When you are the only person using the paperless-ng instance this shouldn't be a problem though. I never tried running paperless with PostgreSQL and therefore cannot tell  what's the improvement gain. I have no need to switch, because I have no problem with the SQLite database so far. Maybe @CorneliousJD can give some insides here?

Edited February 6, 2021 by T0a

[Michelangelo]

1 minute ago, T0a said:

 

I think concurrent access to a file based database such as SQLite is kind of a problem sometimes and results in issues like "database is locked" messages. When you are the only person using the paperless-ng instance this shouldn't be a problem though. I never tried running paperless with PostgreSQL and therefore cannot tell  what's the improvement gain. I have no need to switch, because I have no problem with the SQLite database so far. Maybe @CorneliousJD can give some insides here?

I'm installing it with SQLite as we speak, lets give it a go and see how it works. And yes, i'm the only one so it will probably work good enoug. 

[CorneliousJD]

I've had issues with the "database is locked" for SQLite with other containers and I already had Postgres spun up so I opted to just do it on this from tbe start. 

 

I don't think you'd see any performance improvements until you have hundreds of thousands of documents, or multi users. 

 

Hope this helps!

[Michelangelo]

38 minutes ago, CorneliousJD said:

I've had issues with the "database is locked" for SQLite with other containers and I already had Postgres spun up so I opted to just do it on this from tbe start. 

 

I don't think you'd see any performance improvements until you have hundreds of thousands of documents, or multi users. 

 

Hope this helps!

Thanks for responding! I understand the reason a little bit better now, lets see how it goes with SQLite. 

 

 

All seems to be working perfectly, love it allready. Thanks for making this available in unraid!

 

Two questions:

1) How does file name handling work via unraid config? Is this something i do via the command line or gui? I can't feally find the instructions in the docs, only an explanation on how the principle works. 

I did notice this error/notification in my logs, is it something to worry about?

 

06-02-2021 20:12 WARNING Cannot classify documents: [Errno 2] No such file or directory: '/usr/src/paperless/src/../data/classification_model.pickle'.

 

[Abe677]

On 2/4/2021 at 2:49 PM, daemian said:

I am not 100% sure about this since I don't really use it, and have only been using paperless-ng for a few days now, but the original version of paperless had a defined format for interpreting date, correspondent, title, tags, etc. from the filename. My guess is that paperless-ng uses this same logic.

See here: https://paperless.readthedocs.io/en/latest/guesswork.html

 

If you get your tags and corespondent text matching setup well you should not have to bother. It will find the info in the OCR text from the coument and set those fields along with the data automatically for you. Of course that doesn't always work - but its pretty good most of the time.

 

Yes you are completely correct. I hadn't thought to look at the original project. Thanks!

[Michelangelo]

Over night, the Paperless-ng docker kept my disk busy (you can hear it in this short clip: https://photos.app.goo.gl/bWBroD7MMDjMA19y7) and i've found this error in the logs. Everything in Paperless-Ng seems to be working properly and i'm not sure this error has something do to with it, but i could not find anything related in the logs. And as soon as I turn off paperless docker, the disk stops being busy. 

 

The client mentioned in the errors is my desktop. But i have no connections open to the file folder, Paperless via webbrowser or anything else to unraid. What is it doing? I can't find documentation regarding this error (or i'm looking in the wrong place). Any idea what is happening? 

Feb  6 20:36:04 Unraid nginx: 2021/02/06 20:36:04 [error] 26535#26535: *46095 connect() to unix:/var/tmp/Paperless-ng.sock failed (111: Connection refused) while connecting to upstream, client: 192.168.1.48, server: , request: "GET /dockerterminal/Paperless-ng/ws HTTP/1.1", upstream: "http://unix:/var/tmp/Paperless-ng.sock:/ws", host: "192.168.1.11"
Feb  6 20:36:15 Unraid nginx: 2021/02/06 20:36:15 [error] 26535#26535: *45970 connect() to unix:/var/tmp/Paperless-ng.sock failed (111: Connection refused) while connecting to upstream, client: 192.168.1.48, server: , request: "GET /dockerterminal/Paperless-ng/token HTTP/1.1", upstream: "http://unix:/var/tmp/Paperless-ng.sock:/token", host: "192.168.1.11", referrer: "http://192.168.1.11/dockerterminal/Paperless-ng/"

 

 

edit: forgot to mention that it also makes this sound / keeps busy when all client devices are turned off. Could it be that it keeps the disk busy because it is monitoring the share for new uploads? 

Edited February 7, 2021 by Michelangelo

[Michelangelo]

2 hours ago, Michelangelo said:

Over night, the Paperless-ng docker kept my disk busy (you can hear it in this short clip: https://photos.app.goo.gl/bWBroD7MMDjMA19y7) and i've found this error in the logs. Everything in Paperless-Ng seems to be working properly and i'm not sure this error has something do to with it, but i could not find anything related in the logs. And as soon as I turn off paperless docker, the disk stops being busy. 

 

The client mentioned in the errors is my desktop. But i have no connections open to the file folder, Paperless via webbrowser or anything else to unraid. What is it doing? I can't find documentation regarding this error (or i'm looking in the wrong place). Any idea what is happening? 

Feb  6 20:36:04 Unraid nginx: 2021/02/06 20:36:04 [error] 26535#26535: *46095 connect() to unix:/var/tmp/Paperless-ng.sock failed (111: Connection refused) while connecting to upstream, client: 192.168.1.48, server: , request: "GET /dockerterminal/Paperless-ng/ws HTTP/1.1", upstream: "http://unix:/var/tmp/Paperless-ng.sock:/ws", host: "192.168.1.11"
Feb  6 20:36:15 Unraid nginx: 2021/02/06 20:36:15 [error] 26535#26535: *45970 connect() to unix:/var/tmp/Paperless-ng.sock failed (111: Connection refused) while connecting to upstream, client: 192.168.1.48, server: , request: "GET /dockerterminal/Paperless-ng/token HTTP/1.1", upstream: "http://unix:/var/tmp/Paperless-ng.sock:/token", host: "192.168.1.11", referrer: "http://192.168.1.11/dockerterminal/Paperless-ng/"

 

 

edit: forgot to mention that it also makes this sound / keeps busy when all client devices are turned off. Could it be that it keeps the disk busy because it is monitoring the share for new uploads? 

 

Update on my own post:

 

Looks like i fixed it!

 

For people who might experience the same issue, here is what I did to fix the problem: 

 

Before troubleshooting, I installed Redis via the Community apps platform. I picked the one from Bitnami. I did all configuration, with and without password, and it all worked as expected. But since I don't like the fact that my harddrive is constantly active, i started looking for a solution. 

 

Since I, with the help of jonaswinkler(developer of Paperless-NG) , figured out that the issue was happening somehwere between Paperless and Redis, I removed the Bitnami Redis and installed the Redis docker container manually via the Add container button (used redis:latest to make sure I recieved the latest version). I configured the port, removed the password line from the paperless docker settings and rebooted both dockers. After that, uploading works and the disk stops working after it's done with its tasks. 

 

 

Remaining question: Would you recommend using a password between Redis and Paperless-ng? 

[mweis]

thanks for the container, is working fine for me, but the mailchecker checks and says everything is fine but is not importing any documents. are there any hints what to to? 

[Janddy]

Hi, I want to install paperless-ng but it appears the template is no longer in the CA App store? When I search for paperless I only see the old paperless, and when I search paperless-ng I get nothing. I can see it if I click "get more results from DockerHub" but then the template has no pre-filled variables and paths. Has it been removed for a reason? Or am I doing something wrong?

 

[muppie]

5 hours ago, Janddy said:

Hi, I want to install paperless-ng but it appears the template is no longer in the CA App store? When I search for paperless I only see the old paperless, and when I search paperless-ng I get nothing. I can see it if I click "get more results from DockerHub" but then the template has no pre-filled variables and paths. Has it been removed for a reason? Or am I doing something wrong?

 

I have the same issue and I ran into (my own created) trouble and removed the container with the purpose to re-install and now the template is missing.

[T0a]

The online app store lists it as available (https://unraid.net/community/apps?q=paperless#r). I haven't removed it. So, I have to investigate once I'm home.

 

Update 1: I broke the template with the latest update Should have checked before submitting the changes. Will fix it now and try to get the changes to the CA ASAP

Update 2: I submitted the fix for the template. We wait for getting it merged. Sorry for the trouble guys

Update 3: The template is available again

Edited February 14, 2021 by T0a

[muppie]

5 hours ago, T0a said:

The online app store lists it as available (https://unraid.net/community/apps?q=paperless#r). I haven't removed it.  Have to investigate once I'm home.

 

Update 1: I broke the template with the latest update Should have checked before submitting the changes. Will fix it now and try to get the changes to the CA ASAP

Update 2: I submitted the fix for the template. We wait for getting it merged. Sorry for the trouble guys.

Many thanks!

[muppie]

Hi!

 

Maybe this is a question for the actual developer, but it might not be. I’m having trouble with using environment variables in the unraid template to ensure that the dates are in the local way (DMY) instead of the US way (MDY), but I can’t seem to make it work. I’m attaching the way I have set it up, can someone confirm that it’s correct and it’s wrong somewhere else? All the ”dates created” on the added files in paperless-ng is wrong and based on MDY.

 

Many thanks in advance

 

//Muppie

[mweis]

On 2/10/2021 at 3:57 PM, mweis said:

thanks for the container, is working fine for me, but the mailchecker checks and says everything is fine but is not importing any documents. are there any hints what to to? 

 

here is my mail.log ????

 

[2021-02-15 13:05:20,175] [ERROR] [paperless_mail] Rule 123 Exchange.paperless 123: Error while processing rule: Response status "OK" expected, but "NO" received. Data: [b'[BADCHARSET (US-ASCII)] The specified charset is not supported.']

Traceback (most recent call last):

  File "/usr/src/paperless/src/paperless_mail/mail.py", line 172, in handle_mail_account

    total_processed_files += self.handle_mail_rule(M, rule)

  File "/usr/src/paperless/src/paperless_mail/mail.py", line 214, in handle_mail_rule

    for message in messages:

  File "/usr/local/lib/python3.7/site-packages/imap_tools/mailbox.py", line 100, in fetch

    nums = tuple((reversed if reverse else iter)(self.search(criteria, charset)))[limit_range]

  File "/usr/local/lib/python3.7/site-packages/imap_tools/mailbox.py", line 61, in search

    check_command_status(search_result, MailboxSearchError)

  File "/usr/local/lib/python3.7/site-packages/imap_tools/utils.py", line 54, in check_command_status

    raise exception(command_result=command_result, expected=expected)

imap_tools.errors.MailboxSearchError: Response status "OK" expected, but "NO" received. Data: [b'[BADCHARSET (US-ASCII)] The specified charset is not supported.']