Install worker nodes that can process data, execute bots and pipelines.
Worker nodes are stateless data processing entities that can be installed closed to the source of data generation (ex: on-prem/enterprise/edge sites etc.). Worker nodes execute bots and pipelines and communicate with the RDA platform that is responsible for scheduling and orchestrating jobs (pipelines) across various worker nodes.
With worker nodes you can ingest and process/transform data locally without having to send all the data to centralized locations like an analytics platform or data warehouse. Worker nodes can in one environment can work as a group for load balancing and scale. RDA platform can orchestrate data sharing or routing among workers in distributed environments (ex: workers in edge location exchange data with workers in DC or workers in cloud).
Workers are essentially container nodes and can be installed using Docker or Docker-compose. Workers are typically installed on VMs and are located on-premises/edge environments.
Step-1: Download RDA Fabric Configuration and copy to host where the worker will be installed
Download RDA Fabric Configuration from RDA SaaS portal by going to
Configuration > RDA Configand download the file to the local filesystem where the worker is going to be installed
- Save the file as rda_network_config.json
Download RDA Fabric Configuration
- Create the below directory structure
sudo mkdir -p /opt/rdaf/network_config
sudo mkdir -p /opt/rdaf/worker/config
sudo mkdir -p /opt/rdaf/worker/logs
sudo chown -R `id -u`:`id -g` /opt/rdaf
- Copy the downloaded RDA Fabric configuration file as shown below.
cp rda_network_config.json /opt/rdaf/network_config/rda_network_config.json
- Create common.yml file for logging settings for RDA Worker as shown below.
cat > common.yml << 'EOF'
format: "%(asctime)s %(levelname)s %(module)s - PID=%(process)s %(message)s"
maxBytes: 10485760 # 10MB
handlers: [console, file_handler]
Step-2: Docker Login
Run the below command to create and save the docker login session into CloudFabrix's secure docker repository.
docker login -u='readonly' -p='readonly' cfxregistry.cloudfabrix.io
Step-3: Create Docker Compose File
Create docker compose configuration file for RDA Worker as shown below.
Note: Optionally change the worker group name in the docker-compose file by updating the WORKER_GROUP value. In this example, the worker group name is specified as rda_worker_group01
cat > rda-worker-docker-compose.yml <<EOF
CAPACITY_FILTER: mem_percent < 95
Step-4: Bring Up RDA Worker
docker-compose -f rda-worker-docker-compose.yml pull
docker-compose -f rda-worker-docker-compose.yml up --d
Step-5: Check Worker Status
Check worker node status using
docker pscommand and ensure that worker is up and running, without any restarts. If you see that the worker is restarting, make sure you copied the RDA network config file to the correct location.
docker ps | grep worker
Step-7: Verify New Worker in RDA SaaS portal
A newly installed worker will authenticate with the RDA platform and it will show up in home page summary analytics.
See below before and after comparison of summary analytics. After the worker node is installed it shows up on the home page and Worker Nodes count will increment and the new site will also show up in the Sites section
Step-8: Verify Worker using RDA Client (rdac) utility
rdac pods showing newly installed worker node