Worker Nodes
Install worker nodes that can process data, execute bots and pipelines.
Worker nodes are stateless data processing entities that can be installed closed to the source of data generation (ex: on-prem/enterprise/edge sites etc.). Worker nodes execute bots and pipelines and communicate with the RDA platform that is responsible for scheduling and orchestrating jobs (pipelines) across various worker nodes.
With worker nodes you can ingest and process/transform data locally without having to send all the data to centralized locations like an analytics platform or data warehouse. Worker nodes can in one environment can work as a group for load balancing and scale. RDA platform can orchestrate data sharing or routing among workers in distributed environments (ex: workers in edge location exchange data with workers in DC or workers in cloud).
Workers are essentially container nodes and can be installed using Docker or Docker-compose. Workers are typically installed on VMs and are located on-premises/edge environments.
- Linux OS
- Memory - 8 GB
- Disk - 50 GB
- Python 3.7.4
Step-1: Download RDA Fabric Configuration and copy to host where the worker will be installed
Download RDA Fabric Configuration from RDA SaaS portal by going to
Configuration > RDA Config and download the file to the local filesystem where the worker is going to be installed- Save the file as rda_network_config.json
Download RDA Fabric Configuration
- Create the below directory structure
sudo mkdir -p /opt/rdaf/network_config
sudo mkdir -p /opt/rdaf/worker/config
sudo mkdir -p /opt/rdaf/worker/logs
sudo chown -R `id -u`:`id -g` /opt/rdaf
- Copy the downloaded RDA Fabric configuration file as shown below.
cp rda_network_config.json /opt/rdaf/network_config/rda_network_config.json
- Create common.yml file for logging settings for RDA Worker as shown below.
cd /opt/rdaf/worker/config
cat > common.yml << 'EOF'
version: 1
disable_existing_loggers: false
formatters:
standard:
format: "%(asctime)s %(levelname)s %(module)s - PID=%(process)s %(message)s"
handlers:
console:
class: logging.StreamHandler
level: INFO
formatter: standard
stream: ext://sys.stdout
file_handler:
class: logging.handlers.RotatingFileHandler
level: INFO
formatter: standard
filename: /logs/${pod_type}-${pod_id}.log
maxBytes: 10485760 # 10MB
backupCount: 5
encoding: utf8
root:
level: INFO
handlers: [console, file_handler]
propogate: yes
EOF
Step-2: Docker Login
Run the below command to create and save the docker login session into CloudFabrix's secure docker repository.
docker login -u='readonly' -p='readonly' cfxregistry.cloudfabrix.io
Step-3: Create Docker Compose File
Create docker compose configuration file for RDA Worker as shown below.
Note: Optionally change the worker group name in the docker-compose file by updating the WORKER_GROUP value. In this example, the worker group name is specified as rda_worker_group01
cd /opt/rdaf/worker
cat > rda-worker-docker-compose.yml <<EOF
version: '3.1'
services:
rda_worker:
image: cfxregistry.cloudfabrix.io/ubuntu-rda-worker-all:daily
restart: always
network_mode: host
shm_size: 1gb
volumes:
- /opt/rdaf/network_config:/network_config
- /opt/rdaf/worker/config:/loggingConfigs
- /opt/rdaf/worker/logs:/logs
logging:
driver: "json-file"
options:
max-size: "25m"
max-file: "5"
environment:
RESOURCE_NAME:
RDA_NETWORK_CONFIG: /network_config/rda_network_config.json
LOGGER_CONFIG_FILE: /loggingConfigs/common.yml
WORKER_GROUP: rda_worker_group01
LABELS: name=rda_worker_01
RDA_SELF_HEALTH_RESTART_AFTER_FAILURES: 3
CAPACITY_FILTER: mem_percent < 95
EOF
Step-4: Bring Up RDA Worker
cd /opt/rdaf/worker
docker-compose -f rda-worker-docker-compose.yml pull
docker-compose -f rda-worker-docker-compose.yml up --d
Step-5: Check Worker Status
Check worker node status using
docker ps command and ensure that worker is up and running, without any restarts. If you see that the worker is restarting, make sure you copied the RDA network config file to the correct location. docker ps | grep worker
Step-7: Verify New Worker in RDA SaaS portal
A newly installed worker will authenticate with the RDA platform and it will show up in home page summary analytics.
See below before and after comparison of summary analytics. After the worker node is installed it shows up on the home page and Worker Nodes count will increment and the new site will also show up in the Sites section
Step-8: Verify Worker using RDA Client (rdac) utility
If you have installed RDA Client (rdac) command line utility, you can also verify newly created worker using
rdac pods command. 
rdac pods showing newly installed worker node
Last modified 1yr ago