[Support] uberchuckie - observium


Recommended Posts

@chip The "aria_log_control" should be created in the /config/database volume.

 

/etc/service/mariadb/run is the runit script to start MariaDB in the container. It should be starting MariaDB with

Quote

/usr/bin/mysqld_safe --skip-syslog --datadir='/config/databases'

I'm not sure why it's using /var/lib/mysql in your case.

 

EDIT: Ah, you were running it from the command line. Can you start from scratch and post up the error in the MariaDB log file?

 

The log file should be '/config/databases/8ffb5f7097ea.err'

Edited by uberchuckie
Link to comment

That log file never gets created.

 

/config/databases# dir
aria_log.00000001  ib_buffer_pool  ibdata1            mysql      performance_schema
aria_log_control   ib_logfile0     multi-master.info  observium  test

 

If I run your command above -

/etc/service/mariadb/run
Database exists.
Starting MariaDB...
200924 06:07:37 mysqld_safe Logging to '/config/databases/2e96f60615f2.err'.
200924 06:07:37 mysqld_safe Starting mariadbd daemon with databases from /config/databases

 

service mariadb status
 * MariaDB is stopped.

 

Also I am running Version: 6.8.3 of Unraid.

 

Edited by chip
Link to comment

I may have new news, but it's all a little weird...

 

Because there was some discussion that these issues have to do with mariadb not updating within the docker container, on Friday I went ahead on a whim and ran an apt update and upgrade to all the packages from the observium:latest docker console. Several updates were pulled down -- not sure if that's normal for docker containers, as I generally don't get this far into it :)

 

Following the updates, I started actually getting errors logged to my observium.err file per the reporting in the container log file. The error was:

2020-09-25 20:10:00 0 [ERROR] mysqld: File '/var/log/mysql/mariadb-bin.index' not found (Errcode: 13 "Permission denied")
2020-09-25 20:10:00 0 [ERROR] Aborting

The file in question did exist. Permissions were 660. With permissions changed to 666 or 777 the error persisted, which seemed a little weird. The file in the log seems simply to reference another file in the same directory, mariadb-bin.000001, with the same 660 permissions. I also tried changing those permissions, but it didn't seem to help.

 

I subsequently reverted back to 2.0.2. In writing this post now (~2 days later) I am seeing that in the 2.0.2 container, the files are owned by nobody:users, which is different from the owners that were showing up under the :latest container. In that container, the group was "adm" but I don't remember the user name.

 

Next-level weirdness...I tried to repeat this entire process just now to get the user name again, but I could not reproduce the results -- after running the apt upgrades, things remained in the state where nothing was being logged to observium.err. Worse(?), the mariadb-bin files were not even being created in the container's /var/log/mysql directory.

 

Finally, next-next-level weirdness, I decided I should create a second docker container to mess around with, using observium:latest. So basically I did a new install of the container, with a new container name, into a new appdata folder. When I had reverted back to 2.0.2 above, I had not deleted my apt upgraded :latest image file, so docker used that file to create the new container. And...this one works.

 

Only problem is, all my data is in the 2.0.2 container. I can try to copy my 2.0.2 databases over, but I'd love any recommendations on whether this is a reasonable thing to do, and/or what exactly needs to be copied, and/or how to force a database conversion if one is necessary.

 

With both containers now run-able, I am also happy to run other tests if helpful.

 

Side note, my unRAID Docker page reports that the version is "not available" when I pull the container from uberchuckie/observium or uberchuckie/observium:latest. This is not an issue with uberchuckie/observium:2.0.2. I am slightly nervous there's an updated source on dockerhub and I missed the memo...

Link to comment

@seanwalter MariaDB shouldn't be using /var/log/mysql for its log files or /var/lib/mysql for its data files. It's in the container's ephemeral disk and does not persist between container instances. It's supposed to use the /config/ volume mount.

 

MariaDB is started with /usr/bin/mysqld_safe --skip-syslog --datadir='/config/databases' from the runit service, MariaDB should be using the volume mount for its data.

 

It's weird that a brand new container instance works for you but not @chip. This has got to be something related to file permission issues or something along the line. It's hard for me to debug this without access to a broken system.

Link to comment

I have my settings to /mnt/cache/appdata/observium for all 3 in the container.

 

So with the build as is it can't write to the /config/databases folder so the .err file never shows up.

 

So reading a page back here I ran  -

 

chown -R nobody:users /config/databases

chmod -R 755 /config/databases

 

Then

 

Reran - /usr/bin/mysqld_safe --skip-syslog --datadir='/config/databases'

 

Now I have a .err file in the databases folder.

 

Here is some of that error.

 

2020-09-29 10:32:04 0 [Note] InnoDB: Using Linux native AIO
2020-09-29 10:32:04 0 [Note] InnoDB: Uses event mutexes
2020-09-29 10:32:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-09-29 10:32:04 0 [Note] InnoDB: Number of pools: 1
2020-09-29 10:32:04 0 [Note] InnoDB: Using SSE4.2 crc32 instructions
2020-09-29 10:32:04 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2020-09-29 10:32:04 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2020-09-29 10:32:04 0 [Note] InnoDB: Completed initialization of buffer pool
2020-09-29 10:32:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-09-29 10:32:04 0 [Note] InnoDB: 128 rollback segments are active.
2020-09-29 10:32:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-09-29 10:32:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-09-29 10:32:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-09-29 10:32:04 0 [Note] InnoDB: 10.5.4 started; log sequence number 37755; transaction id 9
2020-09-29 10:32:04 0 [Note] InnoDB: Loading buffer pool(s) from /config/databases/ib_buffer_pool
2020-09-29 10:32:04 0 [Note] Plugin 'FEEDBACK' is disabled.
2020-09-29 10:32:04 0 [Note] InnoDB: Buffer pool(s) load completed at 200929 10:32:04
2020-09-29 10:32:04 0 [Note] Zerofilling moved table:  './mysql/plugin'
2020-09-29 10:32:04 0 [Warning] Checking table:   './mysql/plugin'
2020-09-29 10:32:04 0 [ERROR] mysql.plugin: Table is from another system and must be zerofilled or repaired to be usable on this system
2020-09-29 10:32:04 0 [Warning] Recovering table: './mysql/plugin'
2020-09-29 10:32:04 0 [ERROR] mysql.plugin: 9 when fixing table
2020-09-29 10:32:04 0 [ERROR] Couldn't repair table: mysql.plugin
2020-09-29 10:32:04 0 [ERROR] Could not open mysql.plugin table: "Index for table 'plugin' is corrupt; try to repair it". Some plugins may be not loaded
2020-09-29 10:32:04 0 [Note] Zerofilling moved table:  './mysql/servers'
2020-09-29 10:32:04 0 [Warning] Checking table:   './mysql/servers'
2020-09-29 10:32:04 0 [ERROR] mysql.servers: Table is from another system and must be zerofilled or repaired to be usable on this system

 

 

Any Ideas?

Edited by chip
Link to comment

@chip Can you delete everything and try the 3.0_beta image? I'm not sure why the table would be corrupted right after being created.

 

Edit the container definition and change the Repository from uberchuckie/observium to uberchuckie/observium:3.0_beta.

 

The image contains Observium 20.9, MariaDB 10.5.5 and a change to the run script to always update the file permissions.

 

 

Edited by uberchuckie
Link to comment

Remove the Docker and the folder.

 

Added it and then changed it to uberchuckie/observium:3.0_beta.

I also change the other two settings to /mnt/cache/appdata

 

Thought it was good for a moment as it loaded the log in page. But again DB Error 2002: No such file or directory.

 

Nope looks to be in the same messed up state on my end.

 

Still won't write the log err file to config/databases either so still must be a rights issue again as well.

 

This is odd as all my other dockers work fine so I believe I don't have an issue with my cache drive that I use for dockers.

 

 

Edited by chip
Link to comment

I also ran the following commands after it was still broken -

chown -R nobody:users /config/databases

chmod -R 755 /config/databases

 

No go/changes and log file still doesn't write to /config/databases

 

So I did the following

chmod -R 777 /config/databases

 

Noticed the err log file showed up in /config/databases

 

Skipped looking at the err file and just ran

/usr/bin/mysqld_safe --skip-syslog --datadir='/config/databases'

 

and it is working.

 

I am far from a linux guy just know the basics and know the above commands are about rights/etc but something still seems to be broken in my mind.

Again not a linux expert but something isn't working properly.

 

 

 

Link to comment

OK, I think I know what's going on. When MariaDB first starts with the script to bootstrap itself, it's using the mysql:mysql user/group (104:106). The script changes the ownership to nobody:users (99:100) and causes the file permission issues. This has always been the behaviour and I don't know why this doesn't fail for others but fail for some. It makes sense that changing the permissions to 777 works for you.

 

My host doesn't have user 104 and group 106 defined. Maybe this is why it's working. I'll need to think about how to fix this without breaking it for others.

Link to comment

@chip @seanwalter I think I've fixed it. I have updated the 3.0_beta tag with the fix. If everything goes well, I'll push all the updates to the latest tag.

 

@seanwalter Please backup your data before you try it. The 3.0_beta tag contains Observium 20.9 and would perform an irreversible schema update.

 

Thanks for your patience to help me track down this bug. The pull request for the fix is: https://github.com/charlescng/docker-containers/pull/8

Edited by uberchuckie
Link to comment

@uberchuckie good news...updating my 2.0.2 container to 3.0_beta worked -- my data history is intact, I can connect via the web GUI using my user name and password, etc...

 

It looks like others are having success as well, though to close the loop, I am using /mnt/cache/appdata for my observium config and other files.

 

Thank you for sticking with us on this, and working through the issues!

Link to comment
16 hours ago, seanwalter said:

@uberchuckie good news...updating my 2.0.2 container to 3.0_beta worked -- my data history is intact, I can connect via the web GUI using my user name and password, etc...

 

It looks like others are having success as well, though to close the loop, I am using /mnt/cache/appdata for my observium config and other files.

 

Thank you for sticking with us on this, and working through the issues!

@seanwalter Thanks for helping to debug. You can switch back to the latest tag as all the fixes have been applied to it.

Link to comment
  • 2 weeks later...

Hi uberchuckie, thank you for your time and effort in maintaining this container.
I was wondering if the issue with the timezone did get fixed ? Yesterday i installed the container again and set the TZ in the properties but its still showing in UTC. I searched the thread but didnt find a anwser.

 

Kind regards.

Link to comment
  • 2 weeks later...
On 1/26/2021 at 8:23 AM, Silvan-0172 said:

Hi uberchuckie, thank you for your time and effort in maintaining this container.
I was wondering if the issue with the timezone did get fixed ? Yesterday i installed the container again and set the TZ in the properties but its still showing in UTC. I searched the thread but didnt find a anwser.

 

Kind regards.

Yeah, I don't think anyone has figured it out yet.

Link to comment
  • 2 weeks later...

I created a test image yesterday based on phusion's alpha Ubuntu 20.04 based baseimage. It seems to work from my short testing. The biggest change is upgrading Python to 3.8 from 2.7, the latter of which has reached End-Of-Life awhile back. I've tagged the image as focal-alpha if people want to try it out.

 

Observium and MariaDB doesn't change in this image so there shouldn't be any differences other than upgraded libraries.

 

Again, this is just for testing or if you were curious, I wouldn't upgrade right now if you depend on it working correctly.

Screen Shot 2021-02-16 at 9.36.07 AM.png

Edited by uberchuckie
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.