Event Gateway

Install event gateway RDA agent to collect and stream Module to collect logs/events and stream to RDA platform

Overview

Event Gateway is a type of RDA Agent that can send streaming data to the RDA platform. If a user wants to send logs/events in real-time to the RDA platform, users can install Event Gateway in their local environment and configure event sources to send data to Event Gateway.

Demo Environment sending logs/events to Event Gateway and RDA doing data reduction and routing routing to Splunk, ES

Log Sources: For instance, to send syslogs from your Linux servers to the RDA platform, you can install Event Gateway and configure rsyslog on your Linux servers to send data to Event Gateway, which in turn can send data to the RDA platform.

Existing Log Shippers: Users can also use existing log shippers like Splunk Universal Forwarder, Elasticsearch beats, Fluentd, rsyslog, syslog-ng, etc. to route/send data to Event Gateway and all these are supported as endpoints in Event Gateway.

Installation: Event Gateway runs as a container that can be installed using docker-compose. See here for install instructions

Registration with RDA Platform: The event gateway registers and communicates with the RDA platform using a configuration file that contains your SaaS tenant ID, data fabric access tokens, and object storage credentials. This configuration file can be downloaded from your account in the SaaS portal and specified in the event gateway configuration.

Endpoints: Event Gateway supports endpoints and each endpoint is configured to send data to a stream. For example, you can configure an endpoint with a port and protocol/type (ex: TCP/syslogs) and all syslog sources can send data to that endpoint

Installation Instructions

Prerequisites

• Linux OS

• CPU - 2

• Memory - 8 GB

• Disk - 50 GB

• Python 3.7.4

• Docker version 18.09.2 (or above)

• Docker-compose (1.27.x and above)

Step-1: Download RDA Fabric Configuration and copy to host where Event Gateway will be installed

Download RDA Fabric Configuration from the RDA SaaS portal by going to Configuration > Fabric Configuration and download the file to the local filesystem where the event gateway is going to be installed

• Save the file as rda_network_config.json

Download RDA Fabric configuration

• Create the below directory structure

sudo mkdir -p /opt/rdaf/network_config
sudo mkdir -p /opt/rdaf/event_gateway/config/main
sudo mkdir -p /opt/rdaf/event_gateway/certs
sudo mkdir -p /opt/rdaf/event_gateway/logs
sudo mkdir -p /opt/rdaf/event_gateway/log_archive
sudo chown -R `id -u`:`id -g` /opt/rdaf

• Copy the downloaded RDA Fabric configuration file as shown below.

cp rda_network_config.json /opt/rdaf/network_config/rda_network_config.json

Step-2: Docker Login

Run the below command to create and save the docker login session into CloudFabrix's secure docker repository.

docker login -u='readonly' -p='readonly' cfxregistry.cloudfabrix.io 

Step-3: Create Docker Compose File

Create docker compose configuration file for event gateway as shown below.

Note: Optionally change the agent group name in the docker-compose file by updating the AGENT_GROUP value. In this example, the agent group name is specified as event_gateway_site01

cd /opt/rdaf/event_gateway

cat > event-gateway-docker-compose.yml <<EOF
version: '3.1'
services:
  rda_event_gateway:
    image: cfxregistry.cloudfabrix.io/ubuntu-rda-event-gateway:daily
    restart: always
    network_mode: host
    mem_limit: 6G
    memswap_limit: 6G
    volumes:
    - /opt/rdaf/network_config:/network_config
    - /opt/rdaf/event_gateway/config:/event_gw_config
    - /opt/rdaf/event_gateway/certs:/certs
    - /opt/rdaf/event_gateway/logs:/logs
    - /opt/rdaf/event_gateway/log_archive:/tmp/log_archive
    logging:
      driver: "json-file"
      options:
        max-size: "25m"
        max-file: "5"
    environment:
      RDA_NETWORK_CONFIG: /network_config/rda_network_config.json
      EVENT_GW_MAIN_CONFIG: /event_gw_config/main/main.yml
      AGENT_GROUP: event_gateway_site01
      EVENT_GATEWAY_CONFIG_DIR: /event_gw_config
      LOGGER_CONFIG_FILE: /event_gw_config/main/logging.yml
      RDA_SELF_HEALTH_RESTART_AFTER_FAILURES: 3
    entrypoint: ["/docker-entry-point.sh"]
EOF

Step-4: Bring Up Event Gateway

cd /opt/rdaf/event_gateway

docker-compose -f event-gateway-docker-compose.yml pull 
docker-compose -f event-gateway-docker-compose.yml up --d

Step-5: Check Event Gateway Status

Check event gateway node status using docker ps command and ensure that event gateway is up and running, without any restarts. If you see that the event gateway is restarting, make sure you copied the RDA network config file to the correct location.

docker ps | grep gateway

Step-6: Verify New Event Gateway status in the CFX SaaS portal

A newly installed event gateway will authenticate with RDA Fabric and will show up in the home page summary analytics.

See below for an example. After the event gateway node is installed it shows up on the home page and Agents count will increment and the new site will also show up in the Sites section.

Step-7: Verify Event Gateway using RDA Client (rdac) utility

If you have installed RDA Client (rdac) command line utility, you can also verify the newly created event gateway using rdac agents command.

Event Gateway listed as an RDA agent - Output from rdac agents command

Generating self-signed certificates to enable SSL for the endpoints:

Run the below command on event gateway to generate self-signed certificate files. Fill in the answers for the below prompts.

• Country Name (2 letter code)

• State or Province Name (full name)

• Locality Name (eg, city)

• Organization Name (eg, company)

• Organizational Unit Name (eg, section)

• Common Name (eg, your name or your server's hostname)

• Email Address

openssl req -x509 -newkey rsa:4096 -nodes -out cert.pem -keyout key.pem -days 365

It generates two files under the current working directory, cert.pem and key.pem

Copy the above files to /opt/rdaf/event_gateway/certs directory.

cp cert.pem /opt/rdaf/event_gateway/certs
cp key.pem /opt/rdaf/event_gateway/certs

Endpoints configuration:

RDA event gateway support below end point types.

• Syslog over TCP: Recieve syslog events over TCP protocol

• Syslog over UDP: Recieve syslog events over UDP protocol

• HTTP: Receive log events over HTTP protocol

• TCP: Receive log events over TCP protocol

• Filebeat: Receive log events over HTTP protocol from log shipping agents such as filebeat & winlogbeat

Event gateway with the default configuration for each of the above end points as shown below. The endpoint configuration file is going to be located @ /opt/rdaf/event_gateway/config/endpoint.yml

endpoints:

# Endpoint - Syslog Log events over TCP protocol
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: syslog_tcp_events
  enabled: false
  type: syslog_tcp
  port: 5140
  ssl: false
  ssl_cert_dir: /certs
  attrs:
    site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
    archive_name: syslog_events_archive # Log archive name
  stream: syslog_tcp_event_stream

# Endpoint - Syslog Log events over UDP protocol
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: syslog_udp_events
  enabled: false
  type: syslog_udp
  port: 5141
  attrs:
    site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
    archive_name: syslog_events_archive # Log archive name
  stream: syslog_udp_event_stream

# Endpoint - Events over HTTP protocol
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: http_events
  enabled: false
  type: http
  ssl: false
  ssl_cert_dir: /certs
  content_type: auto
  port: 5142
  attrs:
    site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
    archive_name: http_events_archive # Log archive name
  stream: http_event_stream

# Endpoint - Events in JSON format over TCP protocol
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: tcp_json_events
  enabled: false
  type: tcp_json
  ssl: false
  ssl_cert_dir: /certs
  port: 5143
  attrs:
    site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
    archive_name: tcp_json_events_archive # Log archive name
  stream: tcp_json_event_stream  

# Endpoint - Events from Filebeat agent
# type: filebeat - It is applicable for both Filebeat and Winlogbeat log shipping agents
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: filebeat_events # URL is implicit, http://ip:port/filebeat_events
  type: filebeat
  enabled: false
  ssl: false
  ssl_cert_dir: /certs
  xpack_features: min
  port: 5144
  attrs:
    site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
    archive_name: filebeat_log_events_archive # Log archive name
  stream: filebeat_event_stream

# Endpoint - Windows log events from Winlogbeat agent
# type: filebeat - It is applicable for both Filebeat and Winlogbeat log shipping agents
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: winlogbeat_events # URL is implicit, http://ip:port/winlogbeat_events
  type: filebeat
  enabled: false
  ssl: false
  ssl_cert_dir: /certs
  xpack_features: min
  port: 5145
  attrs:
    site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
    archive_name: winlogbeat_log_events_archive # Log archive name
  stream: winlogbeat_event_stream

For filebeat type endpoint, the supported version of the filebeat and winlogbeat log shipping agent is 7.8.1