Links

Event Gateway

Install event gateway RDA agent to collect and stream Module to collect logs/events and stream to RDA platform

Overview

Event Gateway is a type of RDA Agent that can send streaming data to the RDA platform. If a user wants to send logs/events in real-time to the RDA platform, users can install Event Gateway in their local environment and configure event sources to send data to Event Gateway.
Demo Environment sending logs/events to Event Gateway and RDA doing data reduction and routing routing to Splunk, ES
Log Sources: For instance, to send syslogs from your Linux servers to the RDA platform, you can install Event Gateway and configure rsyslog on your Linux servers to send data to Event Gateway, which in turn can send data to the RDA platform.
Existing Log Shippers: Users can also use existing log shippers like Splunk Universal Forwarder, Elasticsearch beats, Fluentd, rsyslog, syslog-ng, etc. to route/send data to Event Gateway and all these are supported as endpoints in Event Gateway.
Installation: Event Gateway runs as a container that can be installed using docker-compose. See here for install instructions
Registration with RDA Platform: The event gateway registers and communicates with the RDA platform using a configuration file that contains your SaaS tenant ID, data fabric access tokens, and object storage credentials. This configuration file can be downloaded from your account in the SaaS portal and specified in the event gateway configuration.
Endpoints: Event Gateway supports endpoints and each endpoint is configured to send data to a stream. For example, you can configure an endpoint with a port and protocol/type (ex: TCP/syslogs) and all syslog sources can send data to that endpoint

Installation Instructions

Prerequisites

Step-1: Download RDA Fabric Configuration and copy to host where Event Gateway will be installed
Download RDA Fabric Configuration from the RDA SaaS portal by going to Configuration > Fabric Configuration and download the file to the local filesystem where the event gateway is going to be installed
  • Save the file as rda_network_config.json
Download RDA Fabric configuration
  • Create the below directory structure
sudo mkdir -p /opt/rdaf/network_config
sudo mkdir -p /opt/rdaf/event_gateway/config/main
sudo mkdir -p /opt/rdaf/event_gateway/certs
sudo mkdir -p /opt/rdaf/event_gateway/logs
sudo mkdir -p /opt/rdaf/event_gateway/log_archive
sudo chown -R `id -u`:`id -g` /opt/rdaf
  • Copy the downloaded RDA Fabric configuration file as shown below.
cp rda_network_config.json /opt/rdaf/network_config/rda_network_config.json
Step-2: Docker Login
Run the below command to create and save the docker login session into CloudFabrix's secure docker repository.
docker login -u='readonly' -p='readonly' cfxregistry.cloudfabrix.io
Step-3: Create Docker Compose File
Create docker compose configuration file for event gateway as shown below.
Note: Optionally change the agent group name in the docker-compose file by updating the AGENT_GROUP value. In this example, the agent group name is specified as event_gateway_site01
cd /opt/rdaf/event_gateway
cat > event-gateway-docker-compose.yml <<EOF
version: '3.1'
services:
rda_event_gateway:
image: cfxregistry.cloudfabrix.io/ubuntu-rda-event-gateway:daily
restart: always
network_mode: host
mem_limit: 6G
memswap_limit: 6G
volumes:
- /opt/rdaf/network_config:/network_config
- /opt/rdaf/event_gateway/config:/event_gw_config
- /opt/rdaf/event_gateway/certs:/certs
- /opt/rdaf/event_gateway/logs:/logs
- /opt/rdaf/event_gateway/log_archive:/tmp/log_archive
logging:
driver: "json-file"
options:
max-size: "25m"
max-file: "5"
environment:
RDA_NETWORK_CONFIG: /network_config/rda_network_config.json
EVENT_GW_MAIN_CONFIG: /event_gw_config/main/main.yml
AGENT_GROUP: event_gateway_site01
EVENT_GATEWAY_CONFIG_DIR: /event_gw_config
LOGGER_CONFIG_FILE: /event_gw_config/main/logging.yml
RDA_SELF_HEALTH_RESTART_AFTER_FAILURES: 3
entrypoint: ["/docker-entry-point.sh"]
EOF
Step-4: Bring Up Event Gateway
cd /opt/rdaf/event_gateway
docker-compose -f event-gateway-docker-compose.yml pull
docker-compose -f event-gateway-docker-compose.yml up --d
Step-5: Check Event Gateway Status
Check event gateway node status using docker ps command and ensure that event gateway is up and running, without any restarts. If you see that the event gateway is restarting, make sure you copied the RDA network config file to the correct location.
docker ps | grep gateway
Step-6: Verify New Event Gateway status in the CFX SaaS portal
A newly installed event gateway will authenticate with RDA Fabric and will show up in the home page summary analytics.
See below for an example. After the event gateway node is installed it shows up on the home page and Agents count will increment and the new site will also show up in the Sites section.
Step-7: Verify Event Gateway using RDA Client (rdac) utility
If you have installed RDA Client (rdac) command line utility, you can also verify the newly created event gateway using rdac agents command.
Event Gateway listed as an RDA agent - Output from rdac agents command

Generating self-signed certificates to enable SSL for the endpoints:

Run the below command on event gateway to generate self-signed certificate files. Fill in the answers for the below prompts.
  • Country Name (2 letter code)
  • State or Province Name (full name)
  • Locality Name (eg, city)
  • Organization Name (eg, company)
  • Organizational Unit Name (eg, section)
  • Common Name (eg, your name or your server's hostname)
  • Email Address
openssl req -x509 -newkey rsa:4096 -nodes -out cert.pem -keyout key.pem -days 365
It generates two files under the current working directory, cert.pem and key.pem
Copy the above files to /opt/rdaf/event_gateway/certs directory.
cp cert.pem /opt/rdaf/event_gateway/certs
cp key.pem /opt/rdaf/event_gateway/certs

Endpoints configuration:

RDA event gateway support below end point types.
  • Syslog over TCP: Recieve syslog events over TCP protocol
  • Syslog over UDP: Recieve syslog events over UDP protocol
  • HTTP: Receive log events over HTTP protocol
  • TCP: Receive log events over TCP protocol
  • Filebeat: Receive log events over HTTP protocol from log shipping agents such as filebeat & winlogbeat
Event gateway with the default configuration for each of the above end points as shown below. The endpoint configuration file is going to be located @ /opt/rdaf/event_gateway/config/endpoint.yml
endpoints:
# Endpoint - Syslog Log events over TCP protocol
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: syslog_tcp_events
enabled: false
type: syslog_tcp
port: 5140
ssl: false
ssl_cert_dir: /certs
attrs:
site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
archive_name: syslog_events_archive # Log archive name
stream: syslog_tcp_event_stream
# Endpoint - Syslog Log events over UDP protocol
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: syslog_udp_events
enabled: false
type: syslog_udp
port: 5141
attrs:
site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
archive_name: syslog_events_archive # Log archive name
stream: syslog_udp_event_stream
# Endpoint - Events over HTTP protocol
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: http_events
enabled: false
type: http
ssl: false
ssl_cert_dir: /certs
content_type: auto
port: 5142
attrs:
site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
archive_name: http_events_archive # Log archive name
stream: http_event_stream
# Endpoint - Events in JSON format over TCP protocol
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: tcp_json_events
enabled: false
type: tcp_json
ssl: false
ssl_cert_dir: /certs
port: 5143
attrs:
site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
archive_name: tcp_json_events_archive # Log archive name
stream: tcp_json_event_stream
# Endpoint - Events from Filebeat agent
# type: filebeat - It is applicable for both Filebeat and Winlogbeat log shipping agents
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: filebeat_events # URL is implicit, http://ip:port/filebeat_events
type: filebeat
enabled: false
ssl: false
ssl_cert_dir: /certs
xpack_features: min
port: 5144
attrs:
site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
archive_name: filebeat_log_events_archive # Log archive name
stream: filebeat_event_stream
# Endpoint - Windows log events from Winlogbeat agent
# type: filebeat - It is applicable for both Filebeat and Winlogbeat log shipping agents
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: winlogbeat_events # URL is implicit, http://ip:port/winlogbeat_events
type: filebeat
enabled: false
ssl: false
ssl_cert_dir: /certs
xpack_features: min
port: 5145
attrs:
site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
archive_name: winlogbeat_log_events_archive # Log archive name
stream: winlogbeat_event_stream
For filebeat type endpoint, the supported version of the filebeat and winlogbeat log shipping agent is 7.8.1