[Support] Paperless-ng Docker

[mattie112]

Yeah I believe the docker internal DNS only works within the single-network scope (e.g. with docker-compose)

[randonw]

13 minutes ago, muppie said:

Hi!

i have it working but had to define the IP address in upstream_app. 
 

Muppie

 

Where do you define the IP address in the WebUI?

[Lumpy_BD]

Fo what it's worth, my swag and paperless containers are both on the same custom docker network. If I connect to the swag console, I can successfully resolve the IP of the paperless container so network communication *appear* fine between the containers. I'll try it with the IP and see what happens, but I've never had to do this before...

[muppie]

2 minutes ago, randonw said:

 

Where do you define the IP address in the WebUI?

Sorry, i forgot to quote the other post. The one fron Lumpy

[Lumpy_BD]

5 minutes ago, Lumpy_BD said:

Fo what it's worth, my swag and paperless containers are both on the same custom docker network. If I connect to the swag console, I can successfully resolve the IP of the paperless container so network communication *appear* fine between the containers. I'll try it with the IP and see what happens, but I've never had to do this before...

I have tried defining the IP under upstream_app instead of the container name and its still the same... any other ideas?

[randonw]

3 hours ago, mattie112 said:

Do you have a redis container running? And the PAPERLESS_REDIS environment variable correctly set? It uses redis to schedule tasks (like consuming)

 

I have `sameersbn/redis` running with no other config then the port (6379)

 

and the env variable for paperless set to:

redis://iip_unraid_server:6379

 

redis was the answer. Thank you kind person!

[muppie]

16 minutes ago, Lumpy_BD said:

I have tried defining the IP under upstream_app instead of the container name and its still the same... any other ideas?

Did you reboot the swag container?

[Lumpy_BD]

3 minutes ago, muppie said:

Did you reboot the swag container?

Yup.

[Lumpy_BD]

@muppie would you by any chance mind sharing your conf file so I can compare?

[muppie]

I’ll share when I’m back home again in a few hours!

[Steinhose]

Hey,
i get an error message with some documents, but this only happens when they are imported by the consumer. When I upload the document through the gui there was no problems.

These errors happen when umlauts are used, like ö ä ü, these are often used in german.

The error message:
 

[2021-03-23 01:30:44,796] [ERROR] [paperless.consumer] The following error occured while consuming example.pdf: 'utf-8' codec can't encode character '\udcfc' in position 2: surrogates not allowed

 

[muppie]

3 hours ago, Lumpy_BD said:

@muppie would you by any chance mind sharing your conf file so I can compare?

## Version 2020/12/09
# REMOVE THIS LINE BEFORE SUBMITTING: The structure of the file (all of the existing lines) should be kept as close as possible to this template.
# REMOVE THIS LINE BEFORE SUBMITTING: Look through this file for <tags> and replace them. Review other sample files to see how things are done.
# REMOVE THIS LINE BEFORE SUBMITTING: The comment lines at the top of the file (below this line) should explain any prerequisites for using the proxy such as DNS or app settings.
# make sure that your dns has a cname set for <container_name> and that your <container_name> container is not using a base url

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name paperless.*;

    include /config/nginx/ssl.conf;

    client_max_body_size 0;

    # enable for ldap auth, fill in ldap details in ldap.conf
    #include /config/nginx/ldap.conf;

    # enable for Authelia
    include /config/nginx/authelia-server.conf;

    #Organizr ServerAuth
    #include /config/nginx/proxy-confs/organizr-auth.subfolder.conf;
    #auth_request /auth-0;

    location / {
        # enable the next two lines for http auth
        #auth_basic "Restricted";
        #auth_basic_user_file /config/nginx/.htpasswd;

        # enable the next two lines for ldap auth
        #auth_request /auth;
        #error_page 401 =200 /ldaplogin;

        # enable for Authelia
        include /config/nginx/authelia-location.conf;

        include /config/nginx/proxy.conf;
        resolver 127.0.0.11 valid=30s;
        set $upstream_app 192.168.10.90;
        set $upstream_port 8000;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

        # REMOVE THIS LINE BEFORE SUBMITTING: Additional proxy settings such as headers go below this line, leave the blank line above.
    }

}

Please note that I have Authelia enabled so comment those rows if you don’t use it too.

 

Hope it helps!

[Lumpy_BD]

13 hours ago, muppie said:

Please note that I have Authelia enabled so comment those rows if you don’t use it too.

 

Hope it helps!

Yup that did it, thanks so much! they key was the upstream_app like you said... I did try it with the IP address but I had used the internal docker IP (172...) instead of my server's IP (192...). Thanks so much for your help, really appreciated!

Edited March 25, 2021 by Lumpy_BD

[muppie]

19 minutes ago, Lumpy_BD said:

Yup that did it, thanks so much! they key was the upstream_app like you said... I did try it with the IP address but I had used the internal docker IP (172...) instead of my server's IP (192...). Thanks so much for your help, really appreciated!

Great!

[Geow]

Hi all,

This works perfectly after setup, but then when I come back to it the following day it will not consume new documents.

I can see an entry in the paperless-ng logs showing it detected the new file and says it has added it to the task queue, but nothing more happens, unless I restart the container.

Any ideas?

Thanks,

G

 

UPDATE:

It seems the issue is with connecting to Redis. The Paperless-ng container logs are filed with:

[Q] ERROR Error 111 connecting to 10.0.0.90:6379. Connection refused.

 

The Redis container logs show:

 WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

 

but not sure if relevant.

TIA 

Edited March 28, 2021 by Geow
More info

[randalotto]

On 3/26/2021 at 10:55 AM, Geow said:

Hi all,

This works perfectly after setup, but then when I come back to it the following day it will not consume new documents.

I can see an entry in the paperless-ng logs showing it detected the new file and says it has added it to the task queue, but nothing more happens, unless I restart the container.

Any ideas?

Thanks,

G

 

UPDATE:

It seems the issue is with connecting to Redis. The Paperless-ng container logs are filed with:

[Q] ERROR Error 111 connecting to 10.0.0.90:6379. Connection refused.

 

The Redis container logs show:

 WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

 

but not sure if relevant.

TIA 

 

 

I'm having the exact same issue. Here are the log entries from today:

 

[2021-04-01 11:22:35,086] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/src/../consume/scan04012021.pdf to the task queue.
11:22:35 [Q] INFO Enqueued 1
 

Then nothing happens.

Edited April 1, 2021 by randalotto

[muppie]

12 hours ago, randalotto said:

 

 

I'm having the exact same issue. Here are the log entries from today:

 

[2021-04-01 11:22:35,086] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/src/../consume/scan04012021.pdf to the task queue.
11:22:35 [Q] INFO Enqueued 1
 

Then nothing happens.

If you have the same redis connection refused message as @Geow, it doesn’t look like your connection to redis works. Have you checked that?

[Geow]

21 minutes ago, muppie said:

If you have the same redis connection refused message as @Geow, it doesn’t look like your connection to redis works. Have you checked that?

The thing is, the redis connection works initially, and again if you restart the paperless-ng container (no need to restart redis).

[Kevlar75]

Hi,

 

I've been successfully using the original paperless (P) docker for some time and have quite a few correspondents, tags and documents.

 

I have installed paperless-ng (P-ng) and I'm trying to use the importer to setup P-ng with the backup I exported from P.

 

I use the document_importer but because the P json file doesn't include any "file_type" information, I get an exception and nothing imports. 

 

[email protected]:/usr/src/paperless/src# python3.7 manage.py document_importer /usr/src/paperless/export/
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/django/db/models/options.py", line 575, in get_field
    return self.fields_map[field_name]
KeyError: 'file_type'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/django/core/serializers/json.py", line 70, in Deserializer
    yield from PythonDeserializer(objects, **options)
  File "/usr/local/lib/python3.7/site-packages/django/core/serializers/python.py", line 117, in Deserializer
    field = Model._meta.get_field(field_name)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/options.py", line 577, in get_field
    raise FieldDoesNotExist("%s has no field named '%s'" % (self.object_name, field_name))
django.core.exceptions.FieldDoesNotExist: Document has no field named 'file_type'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/usr/src/paperless/src/documents/management/commands/document_importer.py", line 71, in handle
    call_command("loaddata", manifest_path)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 168, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
    self.loaddata(fixture_labels)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 114, in loaddata
    self.load_label(fixture_label)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 172, in load_label
    for obj in objects:
  File "/usr/local/lib/python3.7/site-packages/django/core/serializers/json.py", line 74, in Deserializer
    raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing fixture '/usr/src/paperless/export/manifest.json': 
[email protected]:/usr/src/paperless/src# 

Any help in migrating all my documents and correspondents from Paperless to Paperless-ng would be greatly appreciated.

[Daggr]

On 3/5/2021 at 1:05 AM, lurkio said:

Decided to install and just manually add all my docs from paperless back. I'm getting a strange error on about half of them and google is zero help. 

[2021-03-04 16:55:26,100] [ERROR] [paperless.consumer] Error while consuming document 2021-01-30-0104240000-none-20180222_175754-0000124.jpg: ExifOrientationError: Invalid rotation (0)

The documents in question are absolutely oriented correctly. Has anyone seen this or know of a work around? Thanks!

Got the same error. My documents wont import. Did you manage to solve it?

EDIT: It seems to be only with pictures taken with my phone. If i open the picture with paint and resave it, it works.
If i use the Paperless app, take a picture and crop it. It works.

Edited April 7, 2021 by Daggr
More info

[lurkio]

Sorry Daggr, I never got this resolved. I see your edit, I'll have a look at mine to see if that helps. Thanks

[mattie112]

Does it not work even with that error? I also did see this error a few times but I am not missing anything it seems. But then perhaps it differs from phone to phone?

[Daggr]

13 hours ago, mattie112 said:

Does it not work even with that error? I also did see this error a few times but I am not missing anything it seems. But then perhaps it differs from phone to phone?

No it doesn't work when i get that error.

[mattie112]

5 hours ago, Daggr said:

No it doesn't work when i get that error.

Allright, wel at least there is a work-around for now by cropping

[CvT]

Constant disk activity

 

Just installed the Paperless-ng and Redis dockers and it works, very nice:)

 

My only issue is that the Redis docker produces a constant disk activity (super steady 144 KB/s write, no read activity) at all times. I removed all documents from Paperless-ng but the disk activity still persists. As soon as I stop the Paperless-ng docker the disk activity of the Redis docker stops, so it definitely has something to do with Paperless-ng.

This seems to be a bug, why would Redis need a constant 150KB/s write stream when there is no activity at all in the Redis or Paperless-ng dockers?

I checked the free memory in the Redis docker to see if it isn't swap activity but that's not the case.

Anybody any idea what is happening? This constant disk useless disk activity is preventing me from using paperless-ng at the moment as it will wear out my cache SSD like this.

 

I am using the Redis docker image from A75G's Repository

 

SOLVED: Switched to another Redis docker image from jj9987's Repository and now no more weird disk activity.

 

New problem: Unraid started giving me warnings that the Docker image was quickly filling up with about 1% per 5 minutes. I looked at the container sizes (bottom unraid docker page) and saw that the paperless-ng container was 7GB big... Looking at the paperless-ng log files I saw a huge amount of documents stuck in the tmp folder. So remapped the tmp folder to the appdata folder solved the problem and the paperless-ng container stayed steady at a size just over 1GB after a clean re-install. But the weird thing is that the tmp folder keeps growing and those files are somehow never deleted by paperless-ng (at least not in the 2 days timeframe I watched it). The folder grew to a whopping 7GB again. Is this a bug?

Edited April 12, 2021 by CvT
new problem