Logs & Traces & Metrics

Rierino microservices include configurable logging and monitoring capabilities.

Default Logging

API gateway and microservice layers include Logback logging capabilities out of box. Alternative logging libraries can also be used, by replacing logback dependency in build.gradle files (or ConfigMap entries), if preferred.

The default deployment includes DEBUG level logging to Console with the following pattern:

%d{HH:mm:ss.SSS} [admin-core] [%thread] %-5level %logger{36} - %X - %msg%n

The log level can be defined when deploying new services either through the Deployment UI or helm chart parameters.

The logging pattern and the destination (such as rotating files or SIEM platforms) can easily be configured by editing the logback.xml file (or ConfigMap entry) for the required microservices.

In order to display root cause of errors, which are received in "payload.cause" field from runners, gateway logging level should be set to TRACE. Otherwise, the root cause is hidden to avoid information exposure and only a more generic "message" field is available instead.

OpenTelemetry Support

Rierino uses OpenTelemetry to facilitate monitoring of availability and performance across all platform components. OTLP feeds for Logs, Traces and Metrics can be enabled and configured easily and selectively for each component.

Rierino OpenTelemetry instrumentation combines zero-code Java agent with OpenTelemetry APIs for a mix of automated and manual spans and metrics. This approach provides access to official instrumentations of external system libraries (such as Kafka, MongoDB and Spring) as well as Rierino specific attributes and event details (such as runner or handler specific details).

Instrumentation

Recommended approach for configuring the instrumentation depends on the model used for deploying Rierino:

Kubernetes - Helm Chart Deployments

Official OpenTelemetry helm charts and operators can be used for deploying components required for instrumentation on Rierino kubernetes clusters. Typical installation includes following main activities:

1. Installation of a deployment with collector helm chart to collect cluster metrics and kubernetes events

2. Installation of a daemonset with collector helm chart to collect host & service specific feed

3. Installation of the operator helm chart to manage collectors and auto-instrumentation

4. Creation of "Instrumentation" kind resources with exporter configurations

Details of these activities along with sample chart values can be found below:

Getting StartedOpenTelemetry

OpenTelemetry Kubernetes Set-up Details

OpenTelemetry resources allow auto-instrumentation of services through simple annotations. These annotations are already included as a parameter in gateway & runner helm charts, which is enabled with the "useOTLP" parameter, set to false as default. Setting this parameter value to "true" during deployment automatically configures OTLP instrumentation using OpenTelemetry annotation as follows:

instrumentation.opentelemetry.io/inject-java: {{ .Values.otlpInstrument }}

VM - Sandbox Deployments

Official OpenTelemetry Java Agent can be used for instrumentation of non-kubernetes deployments, such as containerized sandboxes. Typical instrumentation of Rierino Sandbox includes the following main activities:

1. Download opentelemetry-javaagent.jar from OpenTelemetry repository.
2. Copy this jar file to /extra folder inside admin_controller, admin_runner and ops_runner directories on the sandbox VM(s)

3. Set the following environment variable to activate Java agent (either in .env file for all services or in compose.yaml / compose calls for specific microservices):

#/extra contents are automatically copied to /app/datain container.sh
JAVA_TOOL_OPTIONS=-javaagent:/app/data/opentelemetry-javaagent.jar -Dotel.exporter.otlp.endpoint=[URL] -Dotel.exporter.otlp.headers=Authorization="Basic [BASE64]"
#Additional parameters such as the following can be added based on the feed requirements:
#otel.instrumentation.logback-appender.enabled=true 
#otel.instrumentation.common.default-enabled=false
#otel.instrumentation.opentelemetry-api.enabled=true 

Additional Java Agent instrumentation details can be found below:

Getting startedOpenTelemetry

OpenTelemetry Java Agent Details

In addition to excluding certain classes from OpenTelemetry using "otel.javaagent.exclude-classes" parameter, Rierino handlers, state and query managers have a "telemetry.enabled" parameter which can be set to false to supress instrumentation of a specific runner element.

Agent Configuration

OpenTelemetry instrumentation supports regular configurations allowed by the Java Agent using environment variables defined on ConfigMaps or .env files based on the deployment model:

ConfigurationOpenTelemetry

OpenTelemetry Configuration

In standard deployments, iInstrumentation is disabled by default and can be enabled with the following environment variable:

OTEL_INSTRUMENTATION_ENABLED=true

OTLP Feed

OTel configuration requires a target APM platform to export logs, traces and metrics to. For development and testing purposes, this could be "logging". For production systems, this should be an OTLP compatible system.

As an example open-source on-premise option, SigNoz can be used to receive these OTLP feeds. Rierino Ansible playbooks include an example for deploying SigNoz using its official helm charts.

Open source APM | SigNoz

SigNoz OTLP Monitoring

As an example cloud based option. Grafana Cloud can be used to receive these OTLP feeds (e.g. using Alloy inside the cluster or direct feed to Grafana gateway).

Configure Grafana Alloy on Kubernetes | Grafana Alloy documentationGrafana Labs

Grafana Alloy on Kubernetes

Grafana Cloud feeds can be configured using OTel exporter endoint URL configuration (e.g. https://otlp-gateway-prod-[zone].grafana.net/otlp) with authorization header generated using user ID and API key from Grafana account management.

Logs

Easiest way to enable logging with OpenTelemetry is through configuration of a logback appender, by passing "logback.otlp.enabled=true" in JVM arguments, which enables the following section in logback.xml files:

<if condition='"true".equals(property("logback.otlp.enabled"))'>
  <then>
    <appender name="OpenTelemetry" class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
        <numLogsCapturedBeforeOtelInstall>5000</numLogsCapturedBeforeOtelInstall>
    </appender>
    <root level="ERROR">
        <appender-ref ref="OpenTelemetry"/>
    </root>
  </then>
</if>

OpenTelemetry log feeds can be further customized by editing logback.xml files in ConfigMaps for kubernetes deployments and assets under composite folder for sandbox deployments.

In addition to logs displayed on console, OpenTelemetry logging provides better traceability, as event logs registered under span context are automatically correlated with related trace and span identifiers.

OTLP logs are disabled by default and can be enabled with the following environment variable configuration on ConfigMaps or .env files based on the deployment model:

OTEL_LOGS_EXPORTER=otlp
#Or =logging for testing with console output

Traces

All main building blocks of API gateway and backend microservices provide traces for configuration and processing activities. Created spans include key attributes of each event, such as name of the action performed or the ID of the API request. They also include the following linkage for correlation:

• Spans within the same microservice have parent-child relations (i.e. steps in a saga flow are linked with a parent span context)

• Spans triggered by an API gateway call have a (non parent-child) link to the original API gateway span (since microservices can be executing in parallel or in asynch manner as well)

• Front-end systems can send trace and span identifiers to 'define' the originating span, if the API gateway is configured (telemetry.trace.input=true) to accept external span identifiers

Traces are disabled by default and can be enabled with the following environment variable configuration on ConfigMaps or .env files based on the deployment model:

OTEL_TRACES_EXPORTER=otlp
#Or =logging for testing with console output

Metrics

All main building blocks of API gateway and backend microservices provide counter and histogram metrics for all event and role processing actions. Created metrics include key attributes, such as action performed or the result status.

Metrics are disabled by default and can be enabled with the following environment variable configuration on ConfigMaps or .env files based on the deployment model:

OTEL_METRICS_EXPORTER=otlp
#Or =logging for testing with console output