Gateway Services
A gateway service defines a remote Kafka or file system which is used for tracking, logging, and file operations serviced directly by the gateway without involving runners.
Last updated
Core Platform
A gateway service defines a remote Kafka or file system which is used for tracking, logging, and file operations serviced directly by the gateway without involving runners.
Last updated
All services share the following settings:
Type: Type of the service ("kafka" or "fs").
Gateways: List of gateways which are allowed to communicate with this service.
Services also use authentication parameters same as gateway channels, with tokens and access rights defined by path.
Additional parameters applicable for different service types are as follows:
Kafka service is used as a producer, where all Kafka producer settings can be used in parameters.
File system service is used for accessing different file systems using Hadoop libraries, and all Hadoop configurations can be used in parameters.
In addition, Rierino provides 3 customized file system implementations:
(com.rierino.util.fs.FtpFileSystem) is a customization of FTPFileSystem, supporting same fs parameters (e.g. fs.ftp.host, fs.ftp.host.port) with minor improvement on original HDFS FTP file system, caching home directory for faster file listing operations.
(com.rierino.util.fs.CustomS3FileSystem) is a customization of S3AFileSystem, supporting same fs parameters (e.g. fs.s3a.access.key). The main difference is, this implementation automatically adds content-type metadata to uploaded files based on file extensions, allowing simpler use with CDN solutions. The system automatically detects json, pdf, xml, zip, gif, jpg, png, csv, html and txt extensions. It is also possible to override this detection providing fs.s3a.contentTypes parameter with [ext]:[type];[ext]:[type];... key-value pairs.
(com.rierino.util.fs.RestFileSystem) is a file system which uses rest API endpoints for file operations, calling different url paths for actions such as mkdir, rename, upload. Main use case for this file system is integration with CDN solution which do not support other means (e.g. sftp) for uploading files. URL actions that can be configured include:
rename.file
rename.folder
delete.file
delete.folder
mkdir.folder
upload.folder
status.file
status.folder
status.sub.folder
Each of these actions can be configured with the following parameter settings:
rest.[action].path
URL path
http://example.com/rename
-
rest.[action].method
URL method
POST
-
rest.[action].content
Content type
none
application/json
rest.[action].request.pattern
JMESpath pattern for request body
{"data": @}
-
rest.[action].response.pattern
JMESpath pattern for response body
@.content
-
rest.[action].approach
Upload approach (file, data)
data
file
rest.[action].clientPrefix
Prefix to add for rest client configurations
resteasy.
-
rest.header.*, rest.[action].header.*
Headers to add for all & action calls
Authorization=...
-
rest.client.*, rest.[action].client.*
Client properties to add for all & action calls
-
-
All of these custom file systems can be configured in fs definitions using fs.[schema].impl parameter (e.g. fs.s3a.impl = com.rierino.util.fs.CustomS3FileSystem).
If the same system has multiple file services configured for the same schema (e.g. one system mapping s3a to S3AFileSystem, another mapping to CustomS3FileSystem), fs.[schema].impl.disable.cache parameter should be used (e.g. fs.s3a.impl.disable.cache=true), as otherwise all services will use the first cached system class.
