journaldreceiver

Journald Receiver

Status
Stability	alpha: logs
Unsupported Platforms	darwin, windows
Distributions	contrib, k8s
Issues
Code coverage
Code Owners	@belimawr, @namco1992
Emeritus	@djaglowski, @sumo-drosiek

Parses Journald events from systemd journal. Journald receiver requires that:

• the journalctl binary is present in the $PATH of the agent, or is present at journalctl_path if configured; and
• the collector's user has sufficient permissions to access the journal through journalctl.

Configuration

Operators

Each operator performs a simple responsibility, such as parsing a timestamp or JSON. Chain together operators to process logs into a desired format.

• Every operator has a type.
• Every operator can be given a unique id. If you use the same type of operator more than once in a pipeline, you must specify an id. Otherwise, the id defaults to the value of type.
• Operators will output to the next operator in the pipeline. The last operator in the pipeline will emit from the receiver. Optionally, the output parameter can be used to specify the id of another operator to which logs will be passed directly.
• Only parsers and general purpose operators should be used.

Example Configurations

Minimal configuration

The following configuration is the minimal configuration to read journald logs:

receivers:
  journald:

will be passed to journalctl as the following arguments: journalctl ... --priority info. This will read the 10 most recent entries and any subsequent entry. --priority info is the default priority, the following examples will omit it for simplicity.

Cursor tracking

receivers:
  journald:
    storage: file_storage/journald

extensions:
  file_storage/journald:
    directory: .

service:
  extensions: [file_storage/journald]

If you stop and start the otel collector, only new entries will be read.

Reading from the beginning

receivers:
  journald:
    start_at: beginning

will be passed to journalctl as the following arguments: journalctl ... --no-tail. This will read all messages from the current boot.

Units

receivers:
  journald:
    directory: /run/log/journal
    units:
      - ssh
      - kubelet
      - docker
      - containerd
    priority: info

Matches

The following configuration:

- type: journald_input
  matches:
    - _SYSTEMD_UNIT: ssh
    - _SYSTEMD_UNIT: kubelet
      _UID: "1000"

will be passed to journalctl as the following arguments: journalctl ... _SYSTEMD_UNIT=ssh + _SYSTEMD_UNIT=kubelet _UID=1000, which is going to retrieve all entries which match at least one of the following rules:

• _SYSTEMD_UNIT is ssh
• _SYSTEMD_UNIT is kubelet and _UID is 1000

Multiple filtering options

In case of using multiple following options, conditions between them are logically ANDed and within them are logically ORed:

( dmesg )
AND
( priority )
AND
( units[0] OR units[1] OR units[2] OR ... units[U] )
AND
( identifier[0] OR identifier[1] OR identifier[2] OR ... identifier[I] )
AND
( matches[0] OR matches[1] OR matches[2] OR ... matches[M] )
AND
( grep )

Consider the following example:

- type: journald_input
  matches:
    - _SYSTEMD_UNIT: ssh
    - _SYSTEMD_UNIT: kubelet
      _UID: "1000"
  units:
    - kubelet
    - systemd
  priority: info
  identifiers:
    - systemd

The above configuration will be passed to journalctl as the following arguments journalctl ... --priority=info --unit=kubelet --unit=systemd --identifier=systemd _SYSTEMD_UNIT=ssh + _SYSTEMD_UNIT=kubelet _UID=1000, which is going to effectively retrieve all entries which matches the following set of rules:

• _PRIORITY is 6, and

• _SYSTEMD_UNIT is kubelet or systemd, and

• SYSLOG_IDENTIFIER systemd, and

• entry matches at least one of the following rules:

  • _SYSTEMD_UNIT is ssh
  • _SYSTEMD_UNIT is kubelet and _UID is 1000

Setup and deployment

The user running the collector must have enough permissions to access the journal; not granting them will lead to issues.

Docker & Kubernetes

When running otelcol in a container, note that:

1. the container must run as a user that has permission to access the logs
2. the path to the log directory (/run/log/journal, /var/log/journal...) must be mounted in the container
3. depending on your guest system, you might need to explicitly set the log directory in the configuration
4. differences in the systemd version running on the host and on the container may prevent access to logs due to different features and configurations (e.g. zstd compression, keyed hash etc)

Please note that the official otelcol images do not contain the journalctl binary. The next two sections describe two approaches for providing a journalctl binary to the collector when running in a container.

Bundling journalctl with the collector

If the container only needs to run on a limited set of hosts for which there is a common compatible version of journalctl, then you can build a collector container that includes journalctl. There is a simple example with a step-by-step, including a Dockerfile in examples/container.

Invoking journalctl from a mounted chroot

Some collector containers need to work across a variety of arbitrary hosts that may be running mutually incompatible versions of journalctl, making it difficult to select a single version of journalctl to bundle into the container.

One way to solve this problem is to ensure that the collector running in the container invokes the exact same journalctl that was used to write the logs in the first place. To achieve this, you can mount the host's rootfs to the container and then configure the receiver to run the host's journalctl in a chroot for that mount.

You can pass -v /:/host to docker run to mount the host's rootfs:

docker run -v /:/host otel/opentelemetry-collector-contrib

Then, you can configure the receiver with root_path to use that mount as a chroot for journalctl. Due to a Go issue, running executables from $PATH does not work well in a chroot, so you must also use journalctl_path to configure a full path to journalctl inside of the chroot:

receivers:
  journald:
    root_path: /host
    journalctl_path: /usr/bin/journalctl

You also need to ensure the user running the collector can run the journalctl from the chroot, one way to do this is to run the collector as root:

docker run -v /:/host --user 0 otel/opentelemetry-collector-contrib

Linux packaging

When installing otelcol as a linux package, you will most likely need to add the otelcol-contrib or otel user to the systemd-journal group. The exact user and group might vary depending on your package and linux distribution of choice.

You can test if the user has sufficient permissions by running something like (you might need to adjust according to available shell and opentelemetry user)

sudo su -s /bin/bash -c 'journalctl --lines 5' otelcol-contrib

if the permissions are set correctly you will see some logs, otherwise a clear error message.