Overview: Support for vector CA available in the coreylane repo.

Application: Vector - GitHub Project

Docker Hub: https://hub.docker.com/r/timberio/vector/

Template GitHub: https://github.com/coreylane/unraid-ca-templates

 

How do I ingest/collect Unraid logs using Vector? 

Vector can be used with logs of any kind, but my original intention was to use Vector for ingestion of Unraid logs. Please see the below configuration guides for instructions on ingesting Unraid logs with Vector. 

 

Unraid Docker Logs

To ingest Unraid's docker logs using Vector, the container needs read-only access to the unix socket Docker listens on. The default in Unraid is /var/run/docker.sock

 

Add the below path configuration to the Vector container:

 

 

Unraid syslog

To ingest Unraid's system logs using Vector, the container needs read-only access to the syslog file. The default in Unraid is /var/log/syslog

 

Add the below path configuration to the Vector container:

 

 

Vector configuration

When the container paths are setup, the final step is to provide a vector.toml configuration defining the desired log sources and destinations (sinks) where the logs should be shipped. Below is a very simple example Vector configuration that ingests docker logs and unraid syslogs and sends them to the cloud based log service Logtail. 

 

Example /mnt/user/appdata/vector.toml

[sources.docker_logs]
type = "docker_logs"

[sources.unraid_logs]
type = "file"
include = ["/var/log/syslog"]

[sinks.logtail_http]
type = "http"
method = "post"
uri = "https://in.logtail.com/"
encoding.codec = "json"
auth.strategy = "bearer"
auth.token = "XXXXXXXXXXXXXXXXXXXXXX"
inputs = ["docker_logs", "unraid_logs"]

 

New Relic Configuration

The below vector.toml snippit will ship your logs to new relic

 

[sinks.new_relic]
type = "new_relic"
inputs = ["docker_logs", "unraid_logs"]
account_id = "123456"
api = "logs"
license_key = "XXXXXXXXXXXXX"

 

Edited May 19, 20233 yr by coreylane
adding new relic

Hey, my Vector docker has recently stopped working despite no configuration changes.

 

It looks like its failing to recognise the vector.toml configuration file being present and potentially expecting a .yaml or .json version. All three appear valid according to the docs.

https://vector.dev/docs/reference/configuration/

 

Error

2023-12-04T22:36:01.486966Z ERROR vector::config::loading: Config file not found in path. path="/etc/vector/vector.yaml"

 

Thank you for reporting this. I'm having the same issue.

 

This behavior was changed in Vector 0.34. I will work on a fix to the unraid template now.

https://vector.dev/highlights/2023-11-07-0-34-0-upgrade-guide/#default-config-location-change
 

Quote

 

Default config location change

The default config location is now /etc/vector/vector.yaml rather than /etc/vector/vector.toml. This is part of the migration to YAML as Vector’s preferred configuration language (though TOML and JSON will still be supported for the forseeable future). If you need Vector to load /etc/vector/vector.toml you must now specify --config /etc/vector/vector.toml or VECTOR_CONFIG=/etc/vector/vector.toml

 

 

20 hours ago, Steve Croft said:

Error

2023-12-04T22:36:01.486966Z ERROR vector::config::loading: Config file not found in path. path="/etc/vector/vector.yaml"

 

 

Hi Steve, you can fix this by adding a new environment variable to the docker container configuration in Unraid. Please see screenshots below

 

Type: Variable

Key: VECTOR_CONFIG

Value: /etc/vector/vector.toml

 

 

Edited December 5, 20232 yr by coreylane

Thanks @coreylane all sorted! 

I have been really struggling to get it to work with VictoriaLogs - especially since they give all their examples in YAML and it isn't just a 1-to-1 conversion.

Any ideas?

@kesx I don't see VictoriaLogs as a supported source or sink in the documentation https://vector.dev/components/

What does your configuration look like? What errors are you getting, etc? Thanks

OK, I think I finally got syslogs working... Just lots of random moving pieces to fit together just right.

Here is my vector.toml:

[sources.syslog]
type = "file"
include = ["/var/log/syslog"]
ignore_older = 86400  # optional, ignores very old logs
read_from = "beginning"  # or "end"

[transforms.to_syslog]
type = "remap"
inputs = ["syslog"]
source = '''
.message = string!(.message)
'''

[sinks.vlogs]
type = "elasticsearch"
inputs = ["syslog"]
endpoints = ["http://10.0.0.6:9428/insert/elasticsearch/"]
api_version = "v8"
compression = "gzip"

[sinks.vlogs.healthcheck]
enabled = false

[sinks.vlogs.query]
_msg_field = "message"
_time_field = "timestamp"
_stream_fields = ["host", "container_name"]

I'll add docker logs later...