> For the complete documentation index, see [llms.txt](https://docs.rierino.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.rierino.com/devops/microservices/building-blocks/systems-integrations/other-systems.md).

# Other Systems

Other systems range from event streaming to custom Camel based connectors

## CDC

Includes settings required for connecting to a database or a similar system for change data capture. CDC managers produce CDCRecord entries and publish them on a given stream, which can be consumed by a [CDCRoleHandler ](https://docs.rierino.com/devops/microservices/building-blocks/execution-handlers/flow-handlers.md#convert-pulse-to-journal)to convert them into pulse and journal records.

Spring event runners provide support for CDC managers, where each CDC stream linked to a CDC manager can define an offset state (using offset.state parameter of the stream), which is updated based on the specified commit duration (using commitMs parameter of the runner) for managing resume tokens on restart.

{% file src="/files/DZ24UZbWTbypLxkiDRKC" %}
Example Spring Runner with CDC Manager
{% endfile %}

Samza event runners on the other hand, provide more native support for CDC managers, treating them as consumers with input streams with a specific way of configuring access to them.

{% embed url="<https://samza.apache.org/learn/documentation/latest/jobs/configuration-table.html>" %}
Samza Systems Configurations
{% endembed %}

| Samza Setting                                              | Spring Setting   | Definition                                                                                       | Example                               | Default                        |
| ---------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------- | ------------------------------ |
| systems.$alias.consumer.manager                            | manager          | Fully qualified class name for the CDC manager                                                   | com.rierino.state.cdc.MongoCDCManager | -                              |
| systems.$alias.consumer.dlq.suffix                         | dlq.suffix       | Suffix to add to CDC stream names for dead letter queues                                         | \_fail                                | -                              |
| systems.$alias.consumer.dlq.enrich                         | dlq.enrich       | Whether dead letter queue entries should include CDC content                                     | true                                  | false                          |
| systems.$alias.consumer.offset.type                        | offset.type      | Type of resume token / offset value (long, comparable or unordered)                              | long                                  | unordered                      |
| systems.$alias.consumer.pollMs                             | -                | Milliseconds to wait before polling new records                                                  | 5000                                  | 1000                           |
| systems.$alias.consumer.asPulse                            | -                | Whether CDC should produce records as pulse instead of CDC records                               | false                                 | true                           |
| systems.$alias.consumer.manager.parameter.ignoreTerminate  | ignoreTerminate  | Whether the system should stop listening if a TERMINATE operation is received                    | true                                  | false                          |
| systems.$alias.consumer.manager.parameter.onResumeFail     | onResumeFail     | Type of action when CDC manager can not resume from last checkpoint (SKIP, MUTE or FATAL)        | FATAL                                 | SKIP                           |
| systems.$alias.consumer.manager.parameter.onRecordFail     | onRecordFail     | Type of action when CDC manager can not process current change record (SKIP, DLQ, MUTE or FATAL) | FATAL                                 | SKIP                           |
| systems.$alias.consumer.manager.parameter.ignoreResume     | ignoreResume     | Whether the system should ignore current resume token and start as if it is missing              | true                                  | false                          |
| systems.$alias.consumer.manager.parameter.resumeReset      | resumeReset      | Type of strategy to follow on missing resume token (OLDEST or NEWEST)                            | NEWEST                                | OLDEST                         |
| systems.$alias.consumer.manager.parameter.disableReconnect | disableReconnect | Whether reconnecting on failure should be disabled or not                                        | true                                  | false                          |
| systems.$alias.consumer.manager.parameter.retriesPerStep   | retriesPerStep   | Number of reconnect retries on each backoff step                                                 | 3                                     | 1                              |
| systems.$alias.consumer.manager.parameter.backoffSteps     | backoffSteps     | Milliseconds to wait at each backoff step as comma separated values                              | 1000,30000                            | 10,100,200,500,1000,1000,10000 |

In addition to these shared settings, the following CDC managers have additional settings, which are similar to system settings (e.g. systems.$alias.consumer.manager.parameter.uri for MongoDB uri):

* **com.rierino.state.cdc.NoopCDCManager**: Uses "ms" setting for configuring milliseconds to wait between creating a new CDC record with an incremental aggregate ID.
* **com.rierino.state.cdc.ActionCDCManager**: Uses "action" setting for making a call to action path on each iteration and an optional "source.stream" setting for defining source for the action call. Processed event payload can contain 3 main fields:
  * **wait:** If set to true, the action is not triggered till the CDC manager is polled again
  * **offset:** Used as the resume token, which is provided in event payload on the next action call
  * **content:** Stored in content of the produced CDC record
* **com.rierino.state.cdc.MongoCDCManager**: Uses "uri" and "database" settings.
* **com.rierino.state.cdc.RedisCDCManager**: Uses "uri" and "master" settings.
* **com.rierino.state.cdc.EtcdCDCManager**: Uses "url", "namespace", "user", "password" settings.
* **com.rierino.state.cdc.DebeziumCDCManager**: Uses all settings applicable to Debezium connectors.

This manager requires the following dependency added to [deployment contents](https://docs.rierino.com/devops/microservices/deployment-packages/defining-a-deployment.md):

```gradle
implementation (group:'com.rierino.custom', name: 'debezium', version:"${rierinoVersion}")
```

* **com.rierino.state.cdc.HDFSCDCManager**: Uses "uri" and all settings applicable to HDFS file systems for file system change data capture.
* **com.rierino.state.cdc.odata4.OdataCDCManager:** Uses "url" and "path" settings and delta logic of odata v4 endpoints for change data capture.
* **com.rierino.state.cdc.MailCDCManager:** Uses "mail.\*" settings and UID logic of email servers to fetch new emails as change data capture.

{% hint style="info" %}
Runners using CDC managers should be deployed with single replicas since managers consume all records coming from a CDC stream without applying any partitioning. To apply partitioning on these records, the runners should output records to Kafka topics and run business logic on runners consuming these topics.
{% endhint %}

## Custom Stream

A custom stream system uses type `auto`. Its `manager` setting is the fully qualified class name of a custom stream manager. All other settings are passed to that manager. The CDC settings in the preceding table also apply, including `offset.type`, `ignoreResume`, `resumeReset`, `dlq.suffix`, `dlq.enrich`, `onResumeFail`, and `onRecordFail`.

## Kafka

Includes settings required for connecting to a Kafka cluster.

| Setting                           | Definition                                                                                  | Example                                                        | Default |
| --------------------------------- | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | ------- |
| binary                            | Whether the system uses binary or text data format                                          | true                                                           | false   |
| key.class                         | Fully qualified classname for Kafka system record keys                                      | java.lang.String                                               | -       |
| msg.class                         | Fully qualified classname for Kafka system record contents                                  | java.lang.String                                               | -       |
| msg.class.inner                   | Fully qualified inner classname for Kafka system record contents, if msg.class is a generic | java.lang.String                                               | -       |
| rierino.system.$alias.consumer.\* | Kafka consumer configurations (except for Samza)                                            | bootstrap.servers=localhost:9092                               | -       |
| rierino.system.$alias.producer.\* | Kafka producer configurations (except for Samza)                                            | batch.size=1                                                   | -       |
| systems.$alias.\*                 | Samza specific connection configurations                                                    | samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory | -       |
| consumer.\[property]              | Kafka consumer properties                                                                   | auto.offset.reset=earliest                                     | -       |
| producer.\[property]              | Kafka producer properties                                                                   | acks=0                                                         | -       |
| output.backupSystem               | Name of backup system to use if a stream of this system fails                               | kafka\_backup                                                  | -       |
| output.backupStream               | Name of backup stream to use if a stream of this system fails                               | journal\_backup                                                | -       |

{% embed url="<https://kafka.apache.org>" %}
Kafka Page
{% endembed %}

## RabbitMQ

Includes settings required for connecting to Rabbit MQ system.

Connection settings:

| Setting                | Definition                                              | Example                            | Default             |
| ---------------------- | ------------------------------------------------------- | ---------------------------------- | ------------------- |
| uri                    | Full connection uri, overrides host/port/credentials    | amqp\://user:pass\@host:5672/vhost | -                   |
| host                   | Broker host                                             | rabbit-preprod                     | localhost           |
| port                   | Broker port                                             | 5672                               | 5672, 5671 with tls |
| addresses              | Cluster nodes, comma separated, overrides host/port     | node1:5672,node2:5672              | -                   |
| username               | User                                                    | rierino                            | guest               |
| password               | Password                                                | -                                  | guest               |
| vhost                  | Virtual host                                            | /preprod                           | /                   |
| client.id              | Connection name, also the consumer tag on input streams | core-consumer                      | prefix-UUID         |
| client.prefix          | Prefix for default client.id                            | core                               | System alias        |
| connect.wait.ms        | TCP connection timeout                                  | 10000                              | 2000                |
| handshake.timeout.ms   | AMQP handshake timeout                                  | 20000                              | 10000               |
| shutdown.timeout.ms    | Socket close timeout                                    | 5000                               | 10000               |
| channel.rpc.timeout.ms | Timeout for declares and other channel operations       | 30000                              | 10000               |
| heartbeat.sec          | Heartbeat interval, 0 disables                          | 60                                 | 30                  |
| automatic.recovery     | Use client-side recovery instead of the runner's        | true                               | false               |
| topology.recovery      | Redeclare queues and bindings on client-side recovery   | false                              | true                |
| network.recovery.ms    | Client-side retry interval, needs automatic.recovery    | 10000                              | 5000                |

TLS settings:

| Setting                 | Definition                                                                        | Example                     | Default |
| ----------------------- | --------------------------------------------------------------------------------- | --------------------------- | ------- |
| ssl.enabled             | Enable TLS, also flips the default port to 5671                                   | true                        | false   |
| ssl.protocol            | TLS protocol version                                                              | TLSv1.3                     | TLSv1.2 |
| ssl.verify.hostname     | Verify the broker hostname against its certificate                                | false                       | true    |
| ssl.trust.all           | Skip certificate validation, encrypts but authenticates nothing, development only | true                        | false   |
| ssl.truststore.path     | Truststore holding the broker certificate, empty uses the JDK default             | /etc/rierino/truststore.jks | -       |
| ssl.truststore.password | Truststore password                                                               | -                           | -       |
| ssl.truststore.type     | Truststore format                                                                 | PKCS12                      | JKS     |
| ssl.keystore.path       | Client certificate keystore for mutual TLS                                        | /etc/rierino/client.p12     | -       |
| ssl.keystore.password   | Keystore password                                                                 | -                           | -       |
| ssl.keystore.type       | Keystore format                                                                   | PKCS12                      | JKS     |

## STAN

Includes settings required for connecting to a NATS Streaming (NATS) system.

| Setting         | Definition                   | Example | Default                |
| --------------- | ---------------------------- | ------- | ---------------------- |
| nats.url        | NATS URL(s), comma separated | -       | nats\://localhost:4222 |
| cluster.id      | STAN cluster id              | prod    | -                      |
| client.id       | STAN client id               | -       | prefix-UUID            |
| client.prefix   | Prefix for default client.id | core    | System alias           |
| connect.wait.ms | Connection timeout           | 10000   | 2000                   |

## Email

Includes settings required for connecting to an email server system.

| Setting                             | Definition                                                             | Example              | Default |
| ----------------------------------- | ---------------------------------------------------------------------- | -------------------- | ------- |
| mail.transport.protocol             | Protocol used for sending emails (smtp, smtps)                         | smtps                | smtp    |
| mail.smtp.host / mail.smtp.port     | Host and port for sending emails                                       | smtp.gmail.com / 587 | -       |
| mail.smtp.user / mail.smtp.password | Credentials for sending emails                                         | -                    | -       |
| mail.smtp.auth                      | Whether SMTP requires authentication                                   | true                 | false   |
| mail.smtp.starttls.enable           | Whether SMTP connection uses StartTLS                                  | true                 | false   |
| mail.store.protocol                 | Protocol used for receiving emails (imap, imaps, pop3)                 | imaps                | imap    |
| mail.imap.host / mail.imap.port     | Host and port for receiving emails                                     | imap.gmail.com / 993 | -       |
| mail.imap.user / mail.imap.password | Credentials for receiving emails                                       | -                    | -       |
| mail.imap.folder                    | Folder to read emails from                                             | Archive              | INBOX   |
| mail.imap.ssl.enable                | Whether IMAP connection uses SSL                                       | true                 | false   |
| mail.\*                             | Additional Jakarta Mail settings to apply                              | mail.debug=true      | -       |
| mail.rierino.\*                     | Rierino OAuth2Auth authentication settings (when mechanism is XOAUTH2) | -                    | -       |

{% embed url="<https://jakarta.ee/specifications/mail/>" %}
Jakarta Mail
{% endembed %}

This system requires the following dependency added to [deployment contents](https://docs.rierino.com/devops/microservices/deployment-packages/defining-a-deployment.md):

```gradle
implementation (group:'com.rierino.processors', name: 'mail', version:"${rierinoVersion}")
```

## File System

Includes settings required for connecting to a file system. Additional HDFS settings can be applied using site.xml files.

| Setting           | Definition                                                           | Example                                            | Default |
| ----------------- | -------------------------------------------------------------------- | -------------------------------------------------- | ------- |
| uri               | Filesystem root address                                              | hdfs\://localhost:8020/master                      | -       |
| root              | Root folder for paths used by FSEventHandler                         | /uploads                                           | -       |
| fsspec.protocol   | Fsspec protocol when using file system with a Py4J handler           | sftp                                               | -       |
| fsspec.options    | Json representation of fsspec options when using with a Py4J handler | {host:"", port:22, username:"", password:""}       | -       |
| hdfs.\[parameter] | Filesystem parameters when using with an FSEventHandler              | fs.s3a.impl=com.rierino.util.fs.CustomS3FileSystem | -       |

{% hint style="info" %}
Custom file systems listed in "Gateway Services" can be also used with FSEventHandler.
{% endhint %}

{% embed url="<https://hadoop.apache.org>" %}
Hadoop Page
{% endembed %}

When writing to sequence files with FSEventHandler, this system also uses the following settings:

| Setting          | Definition                                                                                    | Example                                             | Default                                             |
| ---------------- | --------------------------------------------------------------------------------------------- | --------------------------------------------------- | --------------------------------------------------- |
| path.writer      | Full class name of the path writer to use for generating file paths                           | com.rierino.handler.util.helper.hdfs.DatePathWriter | com.rierino.handler.util.helper.hdfs.DatePathWriter |
| path.maxRows     | [Maximum records ](#user-content-fn-1)[^1]to include in each sequence file (-1 for unlimited) | 10000                                               | -1                                                  |
| path.bufferSize  | Buffer size for sequence file writer                                                          | 1000                                                | -1                                                  |
| path.blockSize   | Block size for sequence file writer                                                           | 100                                                 | -1                                                  |
| path.compression | Compression to apply on sequence file writer                                                  | BLOCK                                               | NONE                                                |
| path.asBytes     | Whether to write contents as bytes or Text.class                                              | false                                               | true                                                |
| path.format      | Sequence path format to use for DatePathWriter (e.g. one folder per hour)                     | yyyy/MM/dd/hh                                       | yyyy/MM/dd                                          |

## Camel

Includes settings required for connecting to an Apache Camel system.

| Setting    | Definition                        | Example  | Default |
| ---------- | --------------------------------- | -------- | ------- |
| camelRoute | Uri for the Camel system endpoint | mock:out | -       |

{% embed url="<https://camel.apache.org/>" %}
Apache Camel
{% endembed %}

This system requires the following dependency added to [deployment contents](https://docs.rierino.com/devops/microservices/deployment-packages/defining-a-deployment.md):

```gradle
implementation (group:'com.rierino.runner', name: 'camel', version:"${rierinoVersion}")
```

[^1]: File is closed only after this number of records are produced


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.rierino.com/devops/microservices/building-blocks/systems-integrations/other-systems.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
