Worker Nodes

Install worker nodes that can process data, execute bots and pipelines.

Overview

Worker nodes are stateless data processing entities that can be installed closed to the source of data generation (ex: on-prem/enterprise/edge sites etc.). Worker nodes execute bots and pipelines and communicate with the RDA platform that is responsible for scheduling and orchestrating jobs (pipelines) across various worker nodes.

With worker nodes you can ingest and process/transform data locally without having to send all the data to centralized locations like an analytics platform or data warehouse. Worker nodes can in one environment can work as a group for load balancing and scale. RDA platform can orchestrate data sharing or routing among workers in distributed environments (ex: workers in edge location exchange data with workers in DC or workers in cloud).

Workers are essentially container nodes and can be installed using Docker or Docker-compose. Workers are typically installed on VMs and are located on-premises/edge environments.

Prerequisites

• Linux OS

• Memory - 8 GB

• Disk - 50 GB

• Python 3.7.4

• Docker version 18.09.2 (or above)

• Docker-compose (1.27.x and above)

Installation Instructions

Step-1: Download RDA Fabric Configuration and copy to host where the worker will be installed

Download RDA Fabric Configuration from RDA SaaS portal by going to Configuration > RDA Config and download the file to the local filesystem where the worker is going to be installed

• Save the file as rda_network_config.json

Download RDA Fabric Configuration

• Create the below directory structure

sudo mkdir -p /opt/rdaf/network_config
sudo mkdir -p /opt/rdaf/worker/config
sudo mkdir -p /opt/rdaf/worker/logs
sudo chown -R `id -u`:`id -g` /opt/rdaf

• Copy the downloaded RDA Fabric configuration file as shown below.

cp rda_network_config.json /opt/rdaf/network_config/rda_network_config.json

• Create common.yml file for logging settings for RDA Worker as shown below.

cd /opt/rdaf/worker/config

cat > common.yml << 'EOF'
version: 1
disable_existing_loggers: false
​
formatters:
  standard:
    format: "%(asctime)s %(levelname)s %(module)s - PID=%(process)s %(message)s"
​
handlers:
  console:
    class: logging.StreamHandler
    level: INFO
    formatter: standard
    stream: ext://sys.stdout
​
  file_handler:
    class: logging.handlers.RotatingFileHandler
    level: INFO
    formatter: standard
    filename: /logs/${pod_type}-${pod_id}.log
    maxBytes: 10485760 # 10MB
    backupCount: 5
    encoding: utf8
​
root:
  level: INFO
  handlers: [console, file_handler]
  propogate: yes
  
EOF

Step-2: Docker Login

Run the below command to create and save the docker login session into CloudFabrix's secure docker repository.

docker login -u='readonly' -p='readonly' cfxregistry.cloudfabrix.io

Step-3: Create Docker Compose File

Create docker compose configuration file for RDA Worker as shown below.

Note: Optionally change the worker group name in the docker-compose file by updating the WORKER_GROUP value. In this example, the worker group name is specified as rda_worker_group01

cd /opt/rdaf/worker

cat > rda-worker-docker-compose.yml <<EOF
version: '3.1'
services:
  rda_worker:
    image: cfxregistry.cloudfabrix.io/ubuntu-rda-worker-all:daily
    restart: always
    network_mode: host
    shm_size: 1gb
    volumes:
    - /opt/rdaf/network_config:/network_config
    - /opt/rdaf/worker/config:/loggingConfigs
    - /opt/rdaf/worker/logs:/logs
    logging:
      driver: "json-file"
      options:
        max-size: "25m"
        max-file: "5"
    environment:
      RESOURCE_NAME:
      RDA_NETWORK_CONFIG: /network_config/rda_network_config.json
      LOGGER_CONFIG_FILE: /loggingConfigs/common.yml
      WORKER_GROUP: rda_worker_group01
      LABELS: name=rda_worker_01
      RDA_SELF_HEALTH_RESTART_AFTER_FAILURES: 3
      CAPACITY_FILTER: mem_percent < 95

EOF

Step-4: Bring Up RDA Worker

cd /opt/rdaf/worker

docker-compose -f rda-worker-docker-compose.yml pull 
docker-compose -f rda-worker-docker-compose.yml up --d

Step-5: Check Worker Status

Check worker node status using docker ps command and ensure that worker is up and running, without any restarts. If you see that the worker is restarting, make sure you copied the RDA network config file to the correct location.

docker ps | grep worker

Step-7: Verify New Worker in RDA SaaS portal

A newly installed worker will authenticate with the RDA platform and it will show up in home page summary analytics.

See below before and after comparison of summary analytics. After the worker node is installed it shows up on the home page and Worker Nodes count will increment and the new site will also show up in the Sites section

Step-8: Verify Worker using RDA Client (rdac) utility

If you have installed RDA Client (rdac) command line utility, you can also verify newly created worker using rdac pods command.

rdac pods showing newly installed worker node