# Getting Started
Welcome to CloudFabrix Documentation
{% content-ref url="/pages/-M-kfpDIN6k\_B\_mwUKGg" %}
[CloudFabrix Overview](/cfx-overview)
{% endcontent-ref %}
{% content-ref url="/pages/-M-FcyTgkxQd6bqZmiWs" %}
[AIOps Platform](/aiops-platform)
{% endcontent-ref %}
{% content-ref url="/pages/-M-Fd2kZJihm6LYwQawA" %}
[AIOps Solutions](/aiops-solutions)
{% endcontent-ref %}
{% content-ref url="/pages/-MU9NzDBRYc-6NV5snwr" %}
[RDA - Overview](/rda/introduction-to-rda)
{% endcontent-ref %}
{% content-ref url="/pages/-M-l2hJwt7lpY9kzaVVJ" %}
[Observability - IT Infrastructure Monitoring (cfxPulse)](/cfxpulse)
{% endcontent-ref %}
{% content-ref url="/pages/-LyJ4DUYetL6rlefk51m" %}
[Observability - Log Monitoring & Analytics (CFX LogAnalytics or CLA)](/cfxloganalytics)
{% endcontent-ref %}
{% content-ref url="/pages/-Ly8vMwTzwQdr5KgjUEA" %}
[CloudFabrix SaaS](/cfxsaas/signup)
{% endcontent-ref %}
# CloudFabrix Overview
Outcome Driven AIOps For Digital IT Operations And Planning
## Overview
**CloudFabrix** is the leading provider of Outcomes Driven **AIOps Platform and Solutions** to enable AI driven IT Operations and Planning. CloudFabrix enables enterprises to achieve business and operational outcomes through continuous operational and lifecycle insights from data. CloudFabrix solutions are built using flagship **cfxDimensions** - AIOps Platform that leverages AI and Machine Learning to enable customers to solve some of the most pressing challenges IT organizations faces today. CloudFabrix discovers and ingests on-premises and multi-cloud data through 100+ integrations available in the platform.
## CloudFabrix AIOps Platform: cfxDimensions
All CloudFabrix offerings run on the core AIOps platform, called **cfxDimensions**. This is a flagship platform built using cloud native architecture leveraging Microservices architecture and containers for all infrastructure and application services.
#### Highlights of the platform
* AI/ML engine
* Data Ingestion & Integrations
* Cloud Native Architecture
* Microservice Architecture and Containers for infrastructure and application services
* cLambda Serverless engine for high performance & short-lived low-latency services
* Distributed platform: Application runs on multiple VMs
* Both On-prem and SaaS versions
Learn more about [cfxDimensions](https://www.cloudfabrix.com/platform/)
## CloudFabrix AIOps Solutions
CloudFabrix offers two targeted solutions built on cfxDimensions platform. The two solutions are targeted
1. **Asset Intelligence Analytics (AIA)**
2. **Ops Intelligence Analytics (OIA)**
### Asset Intelligence Analytics (AIA) Solution
CloudFabrix **Asset Intelligence Analytics (AIA)**, provides real-time IT asset visibility, utilization and dependencies. It provides key insights about upcoming lifecycle events - End of Life / End of Support, risks due to missing or out-of- contract assets, identify non-compliant assets and measure/track enterprise specific plan of record or compliance policies. This enables IT organizations to not only optimize their existing IT environments, but also effectively plan, track and implement digital transformation initiatives like network refresh with SDN, datacenter consolidation, move to multi-vendor infrastructure etc.
#### AIA covers following key focus areas:
* **360-degree IT Asset Visibility and Intelligence:** End-to-End visibility of all hardware and software assets, using agent-less discovery approach.
* **IT Asset Lifecycle Analytics & Risk Mitigation**: Identification of aging and obsolete assets based on end-of-life (EoL) or end-of-support (EoS) events, identify assets under risk due to no or expiring service contracts, identify assets having security field notices or CVEs.
* **Capacity Planning and Cost Optimization:** Provides asset utilization analytics, like port utilization, license utilization, subscription utilization etc. for enabling capacity planning and mitigation.
* **Compliance Tracking:** Define and track internal compliance policies or plan of record (PoR) - identify number of hardware assets in current generation (N), previous generation (N-1, N-2, N-3 etc.) and software assets having software version that is standard, non-standard or unidentified. Flag all assets that are non-compliant or un-authorized and group by site, DC, service
* **Decision Rooms (coming soon):** Strategize, track and implement IT transformation projects. Perform what-if scenario analysis.
#### **Key Use Cases:**
* 306-degree IT asset visibility and analytics
* IT lifecycle planning and risk mitigation
* IT asset capacity planning
* IT asset compliance tracking and system of record
* DC/Network Refresh or Modernization
* Reduction of AMC costs
* Partner-led Sales Enablement
* Advanced Asset Lifecycle Governance Services
#### Learn more about [AIA](https://www.cloudfabrix.com/solutions/asset-intelligence-and-analytics/).
### Operations Intelligence Analytics (OIA) Solution
CloudFabrix offers **Operations Intelligence Analytics (OIA),** domain-agnostic AIOps solution , to enable IT to evolve into predictive digital IT operations in phases, while realizing immediate benefits with common problems related to event noise and incident response. OIA is available as SaaS and On-Premise deployments.
#### OIA consists of following 3 featured modules:
1. **Alert Watch:** Reduces alert noise and brings actionability to incidents by automatically correlating alerts and events from all monitoring tools.
2. **Incident Room:** Enables rapid diagnosis and resolution of incidents by providing incident-centric and context-aware operational data, knowledge base and task automation,
3. **Stack Watch:** Provides ongoing awareness of alerts, anomalies, and potential issues across the full stack from business components to underlying application and infrastructure components.
#### Learn more about [OIA](https://www.cloudfabrix.com/digital-it-operations/).
### Observability Solution
CloudFabrix offers a set of optional core apps to provide better observability and visibility (enterprise discovery, performance monitoring, log & event monitoring) for environments that have data quality gaps or legacy tools that are ripe for replacement.
* **cfxPulse:** Monitors and analyzes traditional IT, cloud and hybrid environments in near real-time. [Learn more](http://cloudfabrix.com/solutions/hybrid-it-performance-and-availability-monitoring/)
* **cfxDimensions Log Analytics**: Monitor and gain deep insights from millions of logs, flows & events from centralized portal. [Learn more](https://www.cloudfabrix.com/solutions/log-events-and-security-analytics/).
* **IT Optimization Advisor**: Recommendations and Analytics to enable optimization and stabilization of IT infrastructure and operations.
#### Core Apps
* **Data Science App (coming soon)**: Intuitive AI/ML toolkit or a studio for app developers to rapidly prototype, experiment and develop new deep learning models
#### Key Use Cases
Following are some key use cases addressed by OIA solution
* Event aggregation
* Alert noise reduction
* Incident volume reduction
* Incident MTTR reduction
* Accelerating Incident response and remediation
* Visualize full stack dependency mapping
# AIOps Platform
cfxDimensions AIOps Platform
### CloudFabrix AIOps Platform: cfxDimensions
All CloudFabrix offerings run on the core AIOps platform, **cfxDimensions**. This is a flagship platform built in-house using Cloud Native Architecture , leveraging Microservices architecture and containers for all infrastructure and application services.
#### Key Features:
* In-built AI/ML engine.
* Data Ingestion & Integrations.
* Cloud-Native Architecture and Polyglot runtime environment.
* Microservices & Containers for infrastructure and application services.
* cLambda Serverless engine for high performance & short-lived low-latency services.
* Distributed Platform: The Application runs on multiple VMs.
* On-prem and SaaS versions.
#### Learn more about [cfxDimensions](https://www.cloudfabrix.com/platform/)
# AIOps Solutions
CloudFabrix AIOps Solutions - AIA and OIA
CloudFabrix offers two solutions built on cfxDimensions, CloudFabrix's AIOps platform. The two solutions are:
1. **Asset Intelligence Analytics (AIA)**
2. **Ops Intelligence Analytics (OIA)**
### Asset Intelligence Analytics (AIA) Solution
CloudFabrix **Asset Intelligence Analytics (AIA)**, provides real-time IT asset visibility, utilization and dependencies, along with key insights about upcoming lifecycle events (like end of sale/support/life), supportability risks due to missing or out-of- contract assets, identify non-compliant assets and measure/track enterprise specific plan of record or compliance policies. This enables IT organizations to not only optimize their existing IT environments, but also effectively plan, track and implement digital transformation initiatives like network refresh with SDN, datacenter consolidation, move to multi-vendor infrastructure etc.
AIA covers following key focus areas:
* **360-degree IT Asset Visibility and Intelligence:** End-to-end visibility of all hardware and software assets, using agent-less discovery approach
* **IT Asset Lifecycle Analytics & Risk Mitigation**: Identification of aging and obsolete assets based on end-of-life (EoL) or end-of-support (EoS) events, identify assets under risk due to no or expiring service contracts, identify assets having security field notices or CVEs.
* **Capacity Planning and Cost Optimization:** Provides asset utilization analytics, like port utilization, license utilization, subscription utilization etc. for enabling capacity planning and mitigation.
* **Compliance Tracking:** Define and track internal compliance policies or plan of record (PoR) - identify number of hardware assets in current generation (N), previous generation (N-1, N-2, N-3 etc.) and software assets having software version that is standard, non-standard or unidentified. Flag all assets that are non-compliant or un-authorized and group by site, DC, service
* **Decision Rooms (coming soon):** Strategize, track and implement IT transformation projects. Perform what-if scenario analysis.
#### **Key Use Cases:**
* 360-degree IT asset visibility and analytics
* IT lifecycle planning and risk mitigation
* IT asset capacity planning
* IT asset compliance tracking and system of record
* DC/Network Refresh or Modernization
* Reduction of AMC costs
* Partner-led Sales Enablement
* Advanced Asset Lifecycle Governance Services
Lear more about [AIA](https://www.cloudfabrix.com/solutions/asset-intelligence-and-analytics/)
### Operations Intelligence Analytics (OIA) Solution
CloudFabrix offers **Operations Intelligence Analytics (OIA)** domain-agnostic AIOps solution , to enable IT to evolve into predictive digital IT operations in phases, while realizing immediate benefits with common problems related to event noise and incident response. OIA is available as SaaS and On-Prem deployment.
OIA consists of following 3 featured modules
1. **Alert Watch:** Reduces alert noise and brings actionability to incidents by automatically correlating alerts and events from all monitoring tools. Key capabilities include:
2. **Incident Room:** Enables rapid diagnosis and resolution of incidents by providing incident-centric and context-aware operational data, knowledge base and task automation,
3. **Stack Watch:** Provides ongoing awareness of alerts, anomalies, and potential issues across the full stack from business components to underlying application and infrastructure components.
Learn more about [OIA](https://www.cloudfabrix.com/digital-it-operations/)
### Observability Solution
CloudFabrix also offers a set of optional core apps to provide better observability and visibility (enterprise discovery, performance monitoring, log & event monitoring) for environments that have data quality gaps or legacy tools that are ripe for replacement.
* **cfxPulse:** Monitors and analyzes traditional IT, cloud and hybrid environments in near real-time. [Learn more](http://cloudfabrix.com/solutions/hybrid-it-performance-and-availability-monitoring/)
* **cfxDimensions Log Analytics**: Monitor and gain deep insights from millions of logs, flows & events from centralized portal. [Learn more](https://www.cloudfabrix.com/solutions/log-events-and-security-analytics/).
* **IT Optimization Advisor**: Recommendations and Analytics to enable optimization and stabilization of IT infrastructure and operations.
#### Core Apps
* **Data Science App (coming soon)**: Intuitive AI/ML toolkit or a studio for app developers to rapidly prototype, experiment and develop new deep learning models
#### Key Use Cases
Following are some of the key use cases addressed by OIA solution
* Event aggregation
* Alert noise reduction
* Incident volume reduction
* Incident MTTR reduction
* Accelerating Incident response and remediation
* Visualize full stack dependency mapping
# RDA - Overview
CloudFabrix's Robotic Data Automation
**CloudFabrix's** **Robotic Data Automation** **(RDA)** helps enterprises realize value from data faster by simplifying and automating repetitive data ingestion from various sources, preparation and transformation activities with no-code pipelines and recipes with built-in AI/ ML bots, also comes shipped with a range of pre-built AI bots.
With [AIOps Studio](/rda/rda-userguide/rda-aiops-studio), you can author pipelines, explore data bots, visualize data sets, inspect, debug and publish pipelines across your landscape (staging, testing, production, and more).
### **Documentation :**
{% content-ref url="/pages/-MU9Gc8BhoWeWst9fItQ" %}
[RDA - User Guide](/rda/rda-userguide)
{% endcontent-ref %}
{% content-ref url="/pages/-MVAUHRo0V5VfbKq9e1U" %}
[RDA - Data Management (cfxdm)](/rda/rda-userguide/rda-data-management-cfxdm)
{% endcontent-ref %}
{% content-ref url="/pages/-MU9ADnpugqCelNoKr4q" %}
[RDA - AIOps Studio](/rda/rda-userguide/rda-aiops-studio)
{% endcontent-ref %}
{% content-ref url="/pages/-MU8dufjiRUFRet2\_532" %}
[RDA - Administration](/rda/installation-of-rda)
{% endcontent-ref %}
{% content-ref url="/pages/-MUECikQ6RAVZtDgJOnb" %}
[RDA - Installation](/rda/installation)
{% endcontent-ref %}
{% content-ref url="/pages/-MU9LzlZPN3hnnuvC3Sa" %}
[RDA - Python API](/rda/rda-python-api)
{% endcontent-ref %}
{% content-ref url="/pages/-MUFBrAjmhDFdMEg4pPf" %}
[RDA - Datasource Integrations](/rda/cfxdx-datasource-integrations)
{% endcontent-ref %}
{% content-ref url="/pages/-MYK0u\_LHbjbgcmoYC28" %}
[RDA - Terminology and Artifacts](/rda/introduction-to-rda/rda-terminology)
{% endcontent-ref %}
{% content-ref url="/pages/-MU9MCbvvQ82llHHyp1x" %}
[CFXQL - CFX Query Language](/cfxql-cfx-query-language)
{% endcontent-ref %}
# RDA - Terminology and Artifacts
## RDA Terminology
**Solution Packages**: Solution packages are shareable bundles with data, pipelines, and configurations.
**Pipelines**: Pipelines are data operation workflows or Directed Acyclic Graphs(DAGs)
**Bots**: A bot deals with a particular task or a data function. There are three types of pre-fixes to a bot's name.
**#: Bot is source filtered**. It means the data queries and filters are applied while querying the data from datasource.
**\*: Bot is destination filtered**. It means the data queries and filters are applied after retrieving the complete data from the datasource. It is not an efficient approach when dealing with a large amount of data, however, it can be used if the datasource does not support filtering capability while querying the data.
**@: API Endpoint.** This dataset is a wrapper for an API offered by a datasource.
**Dataset**: A dataset holds the data.
**Configurations**: A configuration stores information like credentials, which systems to connect, parameters, etc.
**Plugins**: A plugin defines which system to connect to and its credentials.
## RDA Artifacts:
| Artifact | Description |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Pipelines | A pipeline performs a specific sequence of tasks such as data ingestion, analysing the data, data sanitization, data transformation including applying ML algorithms. |
| Datasets | A dataset is like a dataframe, a saved tabular data. It contains the data. |
| Dictionary Dataset | A dictionary dataset is a dataset used for enrichment of other datasets. |
| Models | A model is a Machine Learning model produced as a result of ML training in a pipeline. |
| Bot Source Configurations | A bot source configuration captures the configuration (if any) and enlists a specific set of bots for automation. |
| Solution Package | A solution package is a bundle of pipelines, datasets, formatting templates, ML models, Bot source configurations etc to accomplish a specific outcome. |
| Data Streams | A data stream is used to notify other pipelines or for exchange of data. Pipelines interacts with other pipelines using datastream. |
| Traces | Traces can be used to track how the data moves through the pipeline. This captures how a pipeline interacts with defined systems, artefacts, etc. |
# RDA - Installation
RDA software download, install and it's prerequisites
### Install and configure the RDA environment
#### **Prerequisites:**
CloudFabrix supports Windows, Linux, and Mac OS environments to install, configure and run **RDA** software. Below are the prerequisites which need to be in place before **RDA** can be installed and configured.
* Windows 10 / Linux / Mac OS X with the below software are installed
* CPU - 2
* Memory - 8 GB
* Disk - 50 GB
* [Docker 18.09.x or above](https://docs.docker.com/engine/install/)
* [Docker Compose](https://docs.docker.com/compose/install/) 1.27.x or above (docker-compose command)
* Python 3.7.4 (recommended) or above
* pip3 utility
* Internet Connectivity to download the '**RDA**' docker image.
* CloudFabrix docker registry URL: cfxregistry.cloudfabrix.io
* If there is HTTP Proxy in place, please refer to Docker documentation on how to configure HTTP Proxy settings for the Docker service.
### **Request RDA software package:**
Please email your request to mentioning the AIOps studio software package.
{% hint style="info" %}
For hosted AIOps studio which requires no installation, fill out the request form at
{% endhint %}
# Linux OS
RDA software installation on Linux OS
{% hint style="info" %}
**Note**: Please check the [Installation](/rda/installation#prerequisites) Prerequisites before you proceed.
{% endhint %}
Run the below commands to verify currently installed RDA prerequisites.
{% hint style="warning" %}
**Note**: Please make sure the below commands are in the PATH variable in the user's login profile.
{% endhint %}
```
docker --version
```
```
docker-compose --version
```
```
python3 --version
```
```
pip3 --version
```
**Step 1**: Request software by contacting . The following procedure assumes you received the download link.
Download the '**rda-docker-compose-with-ssl.tar**' software package.
```
wget https://macaw-amer.s3.amazonaws.com/rda/rda-docker-compose-with-ssl.tar
```
**Step 2**: Extract the '**rda-docker-compose-with-ssl.tar**' software package under the user's home directory or some other directory.
```
tar -xvf rda-docker-compose-with-ssl.tar
```
**Step 3**: Run setup.py python script as shown below.
{% hint style="info" %}
**Note:** The setup.py creates a directory called '**cfx**' under user's home directory for configuration files and exported files. (Ex: /home/macaw/cfx)
{% endhint %}
Below are the '**RDA**' directory locations and their purpose.
**/home/macaw/cfx/cfxdx/config -->** Location of the **conf.yml** file where many datasource credentials and other settings are configured.
**/home/macaw/cfx/cfxdx/output -->** Location of the exported CSV / Excel / JSON files
**Step 4**: To access 'RDA' interface, open up a browser and enter the URL as ****
Note: If firewall service is running, enable port 9998/tcp to access RDA service through a browser.
{% hint style="warning" %}
**Note**: If firewall service is running, enable port 9998/tcp to access RDA service through a browser.
{% endhint %}
**CentOS:**
```
sudo firewall-cmd --add-port=9998/tcp --permanent
```
```
sudo firewall-cmd --reload
```
**Ubuntu:**
This section provides additional steps to deploy RDA on Ubuntu OS (Certified on 18.04).
Using the currently logged-in user, run the following commands to make sure the user has sufficient permissions.
```
macaw@ubuntu-test-box: sudo groupadd docker
macaw@ubuntu-test-box: sudo gpasswd -a $USER docker
```
```
macaw@ubuntu-test-box: docker ps
```
* If the above command throws an error 'permission denied error', run the following command to provide sufficient privileges to the currently logged-in user to run docker (Skip this step if the command does not throw permission denied error or the user has sufficient privileges/permissions).
```
macaw@ubuntu-test-box: sudo chmod 666 /var/run/docker.sock
```
* Run the following commands to install docker-compose for the currently logged-in user.
```
macaw@ubuntu-test-box: sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
macaw@ubuntu-test-box: sudo chmod +x /usr/local/bin/docker-compose
macaw@ubuntu-test-box: sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
```
```
macaw@ubuntu-test-box: sudo ufw allow 9998/tcp
```
{% hint style="info" %}
Note: Access RDA using https\://\:9998/
The default user name is rdademo and the password is rdademo1234
{% endhint %}
{% hint style="info" %}
Note: Default username/password can be changed from docker-compose.yml file under RDA install directory.
{% endhint %}
**Step 4**: Accessing RDA Page
### Install RDA Datanetwork Components (Optional)
The following steps explain how to install Kafka/Zookeeper components needed for RDA Data network bots.\
\
Download [docker-compose](https://macaw-amer.s3.amazonaws.com/rda/rda-components/kafka-zookeeper/docker-compose.yml) file required to install Kafka/Zookeeper and use the below steps to start Kafka/zookeeper docker instances (these instances in turn will be used by RDA runtime)\
\
bash# cd /home/macaw/\
bash# mkdir -p **kafka-zookeeper** \
bash# cd kafka-zookeeper\
bash# wget \\
bash# \\
bash# docker-compose pull\
bash# docker-compose up -d \
\
In addition to the above steps, make sure you enable the following ports
```
sudo firewall-cmd --add-port=2181/tcp --permanent
sudo firewall-cmd --add-port=9092/tcp --permanent
```
```
sudo firewall-cmd --reload
```
### Upgrade RDA
Step 1: Make sure the docker (desktop) environment is intact as per RDA installation prereqs. Also, docker-compose is available to the user (as shown in the below screenshot).
Step 2: Go to the directory where RDA was previously installed.
Step 3: Make sure RDA is up and running. This can be verified by running the docker command as shown in the below screen shot.
Step 4: Go to the directory where RDA is installed and also, docker-compose.yml is available (as shown in the below screenshot).
Step 5: Using the docker-compose command 'docker-compose down' and bring down the RDA instances that are running in your environment as shown in the below screenshot.
Step 6: Using the docker-compose command, upgrade RDA using the 'docker-compose pull' command as shown in the below screenshot.
Step 7: Start RDA upgraded instance using 'docker-compose up -d' as shown in the below screen shot.
Step 8: Verify the RDA docker instances to be up-and-running using the 'docker ps -a' command as shown below screenshot.
Access RDA using **https\://\:9998** and verify the latest version.
###
# Windows OS
RDA software installation on Windows OS
{% hint style="info" %}
**Note**: Please check the [Installation](/rda/installation#prerequisites) Prerequisites before you proceed.
{% endhint %}
Run the below commands to verify currently installed RDA prerequisites.
{% hint style="info" %}
**Note**: Please make sure the below commands are in PATH variable in user's login profile.
{% endhint %}
```
docker --version
```
```
docker-compose --version
```
```
python3 --version
```
```
pip3 --version
```
**Step 1**: Request software by contacting . The following procedure assumes you received the download link.
Download the '**rda-docker-compose-with-ssl.tar**' software package.
**Step 2**: Extract '**rda-docker-compose.tar**' software package and copy the 'cfxdx-docker-compose' folder to C:\ (or other folder location)
{% hint style="info" %}
**Note:** [7zip](https://www.7-zip.org/download.html) or [Winrar](https://www.win-rar.com/download.html?\&L=0) utility is needed to extract the '**cfxdx-docker-compose.tar**' file on Windows OS.
{% endhint %}
**Step 3**: Open Windows Powershell CLI or CMD CLI utility (CLI), go to C:\rda-docker-compose-with-ssl (or other folder location 'rda-docker-compose-with-ssl' install folder is unpacked) and run setup.py python script as shown below.
{% hint style="info" %}
**Note:** The setup.py creates a folder called '**cfx**' under the user's home directory for configuration files and exported files. (Ex: C:\Users\Administrator\cfx)**.**
{% endhint %}
Below are the '**RDA**' folder locations and their purpose.
**C:\Users\Administrator\cfx\cfxdx\config -->** Location of the **conf.yml** file where many datasource credentials and other settings are configured.
**Step 4**: To access the 'RDA' interface, open up a browser and enter the URL as **https\://\:9998**
{% hint style="info" %}
**Note**: If Windows firewall service is enabled, enable ports 9998/tcp to access RDA service through a browser.
{% endhint %}
{% hint style="warning" %}
Note: Access RDA using https\://\:9998/\
\
The default user/password for RDA authentication is: rdademo/rdademo1234
{% endhint %}
### Install RDA Datanetwork Components (Optional)
The following steps explain how to install Kafka/Zookeeper components needed for RDA Data network bots.\
\
Download [docker-compose](https://macaw-amer.s3.amazonaws.com/rda/rda-components/kafka-zookeeper/docker-compose.yml) file required to install Kafka/Zookeeper and use the below steps to start Kafka/zookeeper docker instances (these instances in turn will be used by RDA runtime)
cd C:\\\\
C:\mkdir **kafka-zookeeper** \
C:\cd kafka-zookeeper\
C:\kafka-zookeeper\\\\
C:\kafka-zookeeper\docker-compose pull\
C:\kafka-zookeeper\docker-compose up -d \
\
In addition to the above steps, make sure you enable the following ports are enabled -- 2181, 9092 using Windows firewall commands
### Upgrade RDA
Step 1: Make sure the docker (desktop) environment is intact as per RDA installation prereqs. Also, docker-compose.exe is available to the user (as shown in the below screenshot)
Step 2: Go to the directory where RDA is previously installed.
Step 3: Make sure RDA is up and running. This can be verified by running the docker command as shown in the below screenshot.
Step 4: Go to the directory where RDA is installed and also, docker-compose.yml is available (as shown in the below screenshot).
Step 5: Using the docker-compose command 'docker-compose down' and bring down the RDA instances that are running in your environment as shown in the below screenshot.
Step 6: Using the docker-compose command, upgrade RDA using the 'docker-compose pull' command as shown in the below screenshot.
Step 7: Start RDA upgraded instance using 'docker-compose up -d' as shown in the below screenshot.
Step 8: Verify the RDA docker instances to be up-and-running using the 'docker ps -a' command as shown below screenshot.
Access RDA using **https\://\:9998** and verify the latest version.
# Mac OS
RDA software installation on Mac OS
{% hint style="info" %}
**Note**: Please check the [Installation](/rda/installation#prerequisites) Prerequisites before you proceed.
{% endhint %}
Run the below commands to verify currently installed RDA prerequisites.
{% hint style="warning" %}
**Note**: Please make sure the below commands are in the PATH variable in the user's login profile.
{% endhint %}
```
docker --version
```
```
docker-compose --version
```
```
python3 --version
```
```
pip3 --version
```
**Step 1**: Request software by contacting . The following procedure assumes you received the download link.
Download the '**rda-docker-compose-with-ssl.tar**' software package.
**Step 2:** Create a virtual environment using python3
```
python3 -m venv
eg.
$ python3 -m venv rda-venv
```
**Step 3**: Source virtual env that was created in Step 2 as shown below.
```
$ cd rda-venv
$ source bin/activate
(rda-venv)$
```
**Step 4**: Install docker-compose tool needed to install RDA as shown below
```
(rda-venv)$ pip3 install docker-compose
(rda-venv)$ docker-compose --version
docker-compose version 1.29.0, build 07737305
```
*Note: Make sure docker-compose is installed properly*
\
**Step 5**: Extract the '**rda-docker-compose-with-ssl.tar**' software package under the user's home directory or some other directory.
```
tar -xvf rda-docker-compose-with-ssl.tar
```
**Step 3**: Run setup.py python script as shown below.
{% hint style="info" %}
**Note:** The setup.py creates a directory called '**cfx**' under user's home directory for configuration files and exported files. (Ex: /home/\/cfx)
{% endhint %}
Below are the '**RDA**' directory locations and their purpose.
**/home/\/cfx/cfxdx/config -->** Location of the **conf.yml** file where many datasource credentials and other settings are configured.
**/home/\/cfx/cfxdx/output -->** Location of the exported CSV / Excel / JSON files
**Step 4**: To access 'RDA' interface, open up a browser and enter the URL as ****
{% hint style="info" %}
Note: Access RDA using https\://\:9998/
The default user name is rdademo and the password is rdademo1234
{% endhint %}
{% hint style="info" %}
Note: Default username/password can be changed from docker-compose.yml file under RDA install directory.
{% endhint %}
**Step 4**: Accessing RDA Page
### Install RDA Datanetwork Components (Optional)
The following steps explain how to install Kafka/Zookeeper components needed for RDA Data network bots.\
\
Download [docker-compose](https://macaw-amer.s3.amazonaws.com/rda/rda-components/kafka-zookeeper/docker-compose.yml) file required to install Kafka/Zookeeper and use the below steps to start Kafka/zookeeper docker instances (these instances in turn will be used by RDA runtime)\
\
bash# cd /home/macaw/\
bash# mkdir -p **kafka-zookeeper** \
bash# cd kafka-zookeeper\
bash# wget \\
bash# \\
bash# docker-compose pull\
bash# docker-compose up -d \
\
In addition to the above steps, make sure you enable the following ports
```
sudo firewall-cmd --add-port=2181/tcp --permanent
sudo firewall-cmd --add-port=9092/tcp --permanent
```
```
sudo firewall-cmd --reload
```
###
### Upgrade RDA
Step 1: Make sure the docker (desktop) environment is intact as per RDA installation prereqs. Also, docker-compose is available to the user (as shown in the below screenshot).
Step 2: Go to the directory where RDA was previously installed.
Step 3: Make sure RDA is up and running. This can be verified by running the docker command as shown in the below screenshot.
Step 4: Go to the directory where RDA was installed and also, docker-compose.yml is available (as shown in the below screenshot).
Step 5: Using the docker-compose command 'docker-compose down' and bring down the RDA instances that are running in your environment as shown in the below screenshot.
Step 6: Using the docker-compose command, upgrade RDA using the 'docker-compose pull' command as shown in the below screenshot.
Step 7: Start RDA upgraded instance using 'docker-compose up -d' as shown in the below screenshot.
Access RDA using **https\://\:9998** and verify the latest version.
# RDA Client
Installation instructions for RDA client command line utility to interact with RDA platform
RDA Client (`rdac`) is a command-line utility that allows you to interact with RDA Fabric and RDA Platform. rdac is pre-packaged as a native utility in the following modules. In most cases, you don't need to install rdac separately, but if you need to, jump over to [Installing rdac CLI in your own environment](#iriyoe) (section)
### `rdac` in RDA Studio
1. Open New Terminal from the File menu
2. Type 2 for selecting rdac option
### `rdac` in RDA Worker
If you have installed your own [RDA Worker Nodes](/rda/installation/worker-nodes) for data processing and bot/pipeline execution in your own environment (ex: DC/Edge/On-prem) then you can also use `rdac` that is pre-packaged in the worker node.
1. Find container ID of RDA worker instance using `docker ps`
2. Open interactive shell into the container
3. Use `rdac`
Note however that the usage of rdac in the worker is not recommended for frequent usage, as you would like to minimize interactions and load in the worker node. The preferred option is to use the rdac in Studio (see above)
### Installing rdac CLI in your own environment
Users will be able to use one of the following methods to install/deploy rdac CLI
### **Prerequisites:**
* Python 3.x Environment
* Docker Environment is installed and docker daemon is running
* Curl tool is available
**Step-1: Download the RDA network configuration file**
Download your RDA Credentials and save them under \~/.rda/rda\_network\_config.json (Linux or Mac environments)
**Step-2: Run the curl command to download the RDAC CLI tool**
```
curl -o rdac.py https://bot-docs.cloudfabrix.io/data/wrappers/rdac.py
```
**Step-3: Run the radc command**
```
python rdac.py
```
{% hint style="info" %}
Please refer to [RDA Client CLI Documentation](https://bot-docs.cloudfabrix.io/beginners_guide/rdac/) for more details.
{% endhint %}
# Worker Nodes
Install worker nodes that can process data, execute bots and pipelines.
### Overview
Worker nodes are stateless data processing entities that can be installed closed to the source of data generation (ex: on-prem/enterprise/edge sites etc.). Worker nodes execute bots and pipelines and communicate with the RDA platform that is responsible for scheduling and orchestrating jobs (pipelines) across various worker nodes.
With worker nodes you can ingest and process/transform data locally without having to send all the data to centralized locations like an analytics platform or data warehouse. Worker nodes can in one environment can work as a group for load balancing and scale. RDA platform can orchestrate data sharing or routing among workers in distributed environments (ex: workers in edge location exchange data with workers in DC or workers in cloud).
Workers are essentially container nodes and can be installed using Docker or Docker-compose. Workers are typically installed on VMs and are located on-premises/edge environments.
### Prerequisites
* Linux OS
* Memory - 8 GB
* Disk - 50 GB
* Python 3.7.4
* [Docker version 18.09.2](https://docs.docker.com/engine/install/) (or above)
* [Docker-compose](https://docs.docker.com/compose/install/) (1.27.x and above)
### **Installation Instructions**
**Step-1: Download RDA Fabric Configuration and copy to host where the worker will be installed**
Download RDA Fabric Configuration from RDA SaaS portal by going to `Configuration > RDA Config` and download the file to the local filesystem where the worker is going to be installed
* Save the file as *rda\_network\_config.json*
* Create the below directory structure
```
sudo mkdir -p /opt/rdaf/network_config
sudo mkdir -p /opt/rdaf/worker/config
sudo mkdir -p /opt/rdaf/worker/logs
sudo chown -R `id -u`:`id -g` /opt/rdaf
```
* Copy the downloaded RDA Fabric configuration file as shown below.
* Create common.yml file for logging settings for RDA Worker as shown below.
```
cd /opt/rdaf/worker/config
cat > common.yml << 'EOF'
version: 1
disable_existing_loggers: false
formatters:
standard:
format: "%(asctime)s %(levelname)s %(module)s - PID=%(process)s %(message)s"
handlers:
console:
class: logging.StreamHandler
level: INFO
formatter: standard
stream: ext://sys.stdout
file_handler:
class: logging.handlers.RotatingFileHandler
level: INFO
formatter: standard
filename: /logs/${pod_type}-${pod_id}.log
maxBytes: 10485760 # 10MB
backupCount: 5
encoding: utf8
root:
level: INFO
handlers: [console, file_handler]
propogate: yes
EOF
```
**Step-2: Docker Login**
Run the below command to create and save the docker login session into CloudFabrix's secure docker repository.
```
docker login -u='readonly' -p='readonly' cfxregistry.cloudfabrix.io
```
**Step-3: Create Docker Compose File**
Create docker compose configuration file for **RDA Worker** as shown below.
{% hint style="info" %}
**Note:** Optionally change the worker group name in the docker-compose file by updating the WORKER\_GROUP value. In this example, the worker group name is specified as rda\_worker\_group01
{% endhint %}
```
cd /opt/rdaf/worker
cat > rda-worker-docker-compose.yml < Fabric Configuration` and download the file to the local filesystem where the event gateway is going to be installed
* Save the file as *rda\_network\_config.json*
* Create the below directory structure
```
sudo mkdir -p /opt/rdaf/network_config
sudo mkdir -p /opt/rdaf/event_gateway/config/main
sudo mkdir -p /opt/rdaf/event_gateway/certs
sudo mkdir -p /opt/rdaf/event_gateway/logs
sudo mkdir -p /opt/rdaf/event_gateway/log_archive
sudo chown -R `id -u`:`id -g` /opt/rdaf
```
* Copy the downloaded RDA Fabric configuration file as shown below.
```
cp rda_network_config.json /opt/rdaf/network_config/rda_network_config.json
```
**Step-2: Docker Login**
Run the below command to create and save the docker login session into CloudFabrix's secure docker repository.
```
docker login -u='readonly' -p='readonly' cfxregistry.cloudfabrix.io
```
**Step-3: Create Docker Compose File**
Create docker compose configuration file for event gateway as shown below.
{% hint style="info" %}
**Note:** Optionally change the agent group name in the docker-compose file by updating the AGENT\_GROUP value. In this example, the agent group name is specified as `event_gateway_site01`
{% endhint %}
```
cd /opt/rdaf/event_gateway
cat > event-gateway-docker-compose.yml <
# stream:
- name: syslog_tcp_events
enabled: false
type: syslog_tcp
port: 5140
ssl: false
ssl_cert_dir: /certs
attrs:
site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
archive_name: syslog_events_archive # Log archive name
stream: syslog_tcp_event_stream
# Endpoint - Syslog Log events over UDP protocol
# attrs:
# stream:
- name: syslog_udp_events
enabled: false
type: syslog_udp
port: 5141
attrs:
site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
archive_name: syslog_events_archive # Log archive name
stream: syslog_udp_event_stream
# Endpoint - Events over HTTP protocol
# attrs:
# stream:
- name: http_events
enabled: false
type: http
ssl: false
ssl_cert_dir: /certs
content_type: auto
port: 5142
attrs:
site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
archive_name: http_events_archive # Log archive name
stream: http_event_stream
# Endpoint - Events in JSON format over TCP protocol
# attrs:
# stream:
- name: tcp_json_events
enabled: false
type: tcp_json
ssl: false
ssl_cert_dir: /certs
port: 5143
attrs:
site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
archive_name: tcp_json_events_archive # Log archive name
stream: tcp_json_event_stream
# Endpoint - Events from Filebeat agent
# type: filebeat - It is applicable for both Filebeat and Winlogbeat log shipping agents
# attrs:
# stream:
- name: filebeat_events # URL is implicit, http://ip:port/filebeat_events
type: filebeat
enabled: false
ssl: false
ssl_cert_dir: /certs
xpack_features: min
port: 5144
attrs:
site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
archive_name: filebeat_log_events_archive # Log archive name
stream: filebeat_event_stream
# Endpoint - Windows log events from Winlogbeat agent
# type: filebeat - It is applicable for both Filebeat and Winlogbeat log shipping agents
# attrs:
# stream:
- name: winlogbeat_events # URL is implicit, http://ip:port/winlogbeat_events
type: filebeat
enabled: false
ssl: false
ssl_cert_dir: /certs
xpack_features: min
port: 5145
attrs:
site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
archive_name: winlogbeat_log_events_archive # Log archive name
stream: winlogbeat_event_stream
```
{% hint style="info" %}
`For`` `**`filebeat`**` ``type endpoint, the supported version of the filebeat and winlogbeat log shipping agent is 7.8.1`
{% endhint %}
# Edge Collector
Discover and collect IT assets data from hybrid cloud environments and send data to RDA Platform
## Overview
An Edge Collector is a type of RDA Agent that can discover and collect IT asset data in an agentless manner and send this data to the RDA platform. If a user wants to send data from their network devices (Chassis, Fabric Extender, Interfaces) to the RDA platform, users can install Edge collector in their local environment. Currently, Edge Collector can collect data using **SNMP** and **SSH** protocols.
## Prerequisites
* Linux OS
* Memory - 8 GB
* Disk - 50 GB
* Python 3.7.4
* [Docker version 18.09.2](https://docs.docker.com/engine/install/) (or above)
* [Docker-compose](https://docs.docker.com/compose/install/) (1.27.x and above)
## Installation Instructions
#### Step-1: Download RDA Fabric Configuration and copy to host where Edge Collector Agent will be installed
Download RDA Fabric Configuration from the RDA SaaS portal by going to:
Configuration > Fabric Configuration and download the file to the local filesystem where the Edge Collector Agent is going to be installed
* Save the file as `rda_network_config.json`
Download RDA Fabric configuration
* Copy the downloaded file to `~/network_config/rda_network_config.json`
```
mkdir -p ~/network_config
cp rda_network_config.json ~/network_config/rda_network_config.json
```
#### Step-2: Docker Login
`$ docker login -u=readonly -p='readonly' cfxregistry.cloudfabrix.io`
`WARNING! Using --password via the CLI is insecure. Use --password-stdin. WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store`
#### Step-3: Docker Compose File Download
Download `edge-collector-docker-compose.yml` file from the following location
#### Step-4: Create a credential file for accessing agent to perform discovery
```
bash# mkdir -p ~/cfxedgecollector
bash# mkdir -p ~/cfxedgecollector/cred
```
create `credentials.json` file and enter credentials as shown below in the file and copy file in the directory `~/cfxedgecollector/cred/credentials.json`
We need to add all required credentials for collection in the credentials.json file
`bash# cat credentials.json`
```
[
{
"ipAddress" : [
"10.95.158.*",
"10.95.123.10-10.95.123.20"
],
"id": "",
"type": "device-snmp-v1v2",
"credentials": {
"readCommunity": "*****",
"protocol": "snmpv2c",
"port": 161
}
},
{
"ipAddress" : [
"10.95.158.*"
],
"id": "",
"type": "device-host-ssh",
"credentials": {
"username": "cfxuser",
"password": "*****",
"port": 22,
"protocol": "SSHV2",
"pkey": null,
"passphrase": null
}
},
{
"ipAddress" : [
"10.95.158.9,10.95.158.10"
],
"id": "",
"type": "device-host-ssh",
"credentials": {
"username": "cfxuser",
"password": "*****",
"port": 22,
"protocol": "SSHV2",
"pkey": null,
"passphrase": null
}
}
]
```
**Note:** In the above `credentials.json` file,
* `“id”` field indicates that you can give any unique name for the credential identifier,
* `“type”` field is a standard type used for SNMP and SSH Devices. Supported types are
* `“device-snmp-v1 or device-snmp-v2”` for **SNMP** data collection
* `"device-host-ssh"` for **SSH** data collection for the network devices
#### Step-5: Edit the `edgecollector-agent-docker-compose.yml` file and provide the name of the agent in the last line:
`vi edgecollector-agent-docker-compose.yml (or use your favorite editor to edit / update the file )`
```
rda_edgecollector_agent:
image: 'cfxregistry.cloudfabrix.io/cfxcollector:3.0.0'
restart: always
ports:
- '8889:5000'
volumes:
- '~/network_config:/network_config'
- '~/cfxedgecollector:/cfxedgecollector'
- '~/cfxedgecollector/cred:/cred'
environment:
RDA_NETWORK_CONFIG: /network_config/rda_network_config.json
PYTHONPATH: /opt/cfxedgecollector/
container_name: rda_edgecollector_agent
ulimits:
nproc:
soft: 64000
hard: 128000
nofile:
soft: 64000
hard: 128000
entrypoint:
- /bin/bash
- '-c'
- >-
cd /opt/cfxedgecollector/src/; python -c 'import edgecollector_rda_agent ;
edgecollector_rda_agent.run()' --creddir /cred/ --agent-group-name
```
#### Step-6: Bring Up Edge Collector Agent:
`docker-compose -f edgecollector-agent-docker-compose.yml up -d`
#### Step-7: Check Edge Collector Agent Status:
Check Edge Collector Agent status using `docker p`s command and ensure that Edge Collector Agent is up and running, without any restarts. If you see that the Edge Collector Agent is restarting, make sure you copied the RDA network config file to the correct location.
```
[macaw@sysloglinux110 ~]$ docker ps | grep rda_edgecollector_agent
0be4f22ba07e cfxcollector:cfxcollector-1.0.1 "/bin/bash -c 'cd /o…" 2 days ago Up 2 days 0.0.0.0:8889->5000/tcp rda_edgecollector_agent
[macaw@sysloglinux110 ~]$
```
#### Step-8: Verify Edge Collector Agent in RDA Studio or rdac utility
Command: **!rdac agent-bots**
If you have installed an RDA Client (rdac) command-line utility, you can also verify the newly created Edge Collector Agent using rdac agents command.
Command: **rdac agent-bots**
\
# Log Shippers
Log Shippers Configuration
Log Shippers are the components that send *logs (or log files )* from a file-based data source to a supported output destination. A *Log shipper* is a tool that functions as the glue between your system and RDA, making it possible to transfer log files and metrics easily and reliably to the configured destination.
# Filebeat
Configuration of log shipper 'Filebeat'
This section explains how you can configure 'Filebeat' like a log shipper.
In order for filebeat component to send the log details to the event gateway, users have to configure two elements.
1. Event Gateway Endpoint
2. Filebeat configuration
**Step 1:**
An example of Event Gateway Endpoint configuration is captured in the below configuration snippet
```
endpoints:
- name: fb1 # URL is implicit, http://ip:port/fb1
type: filebeat
stream: filebeat_1_logs
ssl: true
enabled: true
xpack_features: min
attrs:
site_code: dataccenter2
archive_name: filebeat_logs
port: 9200
```
An example of Linux-based Filebeat configuration is captured in the below configuration snippet.
**Step 2:** Update hosts details in /etc/filebeat/filebeat.yml file (using your favorite editor (e.g. vi )
```
output.elasticsearch:
# Boolean flag to enable or disable the output module.
enabled: true
# Array of hosts to connect to.
# Scheme and port can be left out and will be set to the default (http and 9200)
# In case you specify and additional path, the scheme is required: http://localhost:9200/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
hosts: ["http://:9200/fb1"]
protocol: "http"
ssl.enabled: false
#ssl.verification_mode: none
```
**Step 3:** Restart the filebeat service (as shown in the below code snippet).
```
# Restart filebeat services
bash# sudo systemctl stop filebeat
bash# sudo systemctl start filebeat
```
Note: In order to run the above commands, the user is expected to have 'sudo' privileges or run the command as a root to enable the required ports.
# Fluentd
Configuration of log shipper 'Fluentd'
This section explains how you can configure 'Fluentd' like a log shipper.
In order for the Fluentd component to send the log details to the event gateway, users have to configure two elements.
1. Event Gateway Endpoint
2. Fluentd configuration
**Step 1:**
An example Event Gateway Endpoint configuration is captured in the below configuration snippet
```
Gateway Endpoint:
endpoints:
- name: http_events
enabled: true
type: http
content_type: auto
port: 516
attrs:
site_code: cfx_dc3
archive_name: http_events
stream: http-stream-01
```
An example Fluentd configuration is captured in the below configuration snippet.
**Step 2:**
Update host details in /etc/td-agent/td-agent.conf (using your favorite editor (e.g. vi )
```
## CFX Configuration
@type tail
path /var/log/messages
pos_file /var/log/td-agent/log-messages.pos
tag cfx-test-log
@type none
@type http
endpoint http://:516/http_events
open_timeout 2
@type json
flush_interval 10s
```
**Step 3:**
Restart Fluentd service (example is captured in the below snippet)
```
# Restart td-agent services
sudo systemctl stop td-agent
sudo systemctl start td-agent
```
Note: In order to run the above commands, the user is expected to have 'sudo' privileges or run the command as a root to enable the required ports.
# Rsyslog
Configuration of log shipper 'rsyslog'.
This section explains how you can configure 'rsyslog' like a log shipper.
In order for rsyslog component to send the log details to the event gateway, users have to configure two elements.
1. Event Gateway Endpoint
2. Systems rsyslog configuration on Linux environments
**Step 1:**
An example Event Gateway Endpoint configuration is captured in the below configuration snippet.
```
// Gateway Endpoint:
endpoints:
- name: syslog_tcp_events
enabled: true
type: syslog_tcp
port: 514
attrs:
site_code: cfx_dc1
archive_name: network_syslogs
stream: syslog-tcp-stream-01
```
An example Linux Service configuration is captured in the below configuration snippet.
**Step 2:** Uncomment below lines in /etc/rsyslog.conf using your favorite editor (e.g. vi )
```
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")
module(load="imtcp") # needs to be done just once
input(type="imtcp" port="514")
#Target="remote_host" Port="XXX" Protocol="tcp")
*.* @@:514 # Use @@ for TCP protocol
```
**Step 3:** Enable the required firewall-ports (514 in this case for tcp/udp) using respective commands (An example of CentOS/RHEL based commands are captured below snippet).
```
bash# sudo firewall-cmd --add-port=514/tcp --permanent
bash# sudo firewall-cmd --add-port=514/udp --permanent
bash# sudo firewall-cmd --reload
```
**Step 4:** Restart the rsyslog service (An example of CentOS/RHEL based commands are captured below snippet).
> bash# sudo systemctl restart rsyslog
Note: In order to run the above commands, the user is expected to have 'sudo' privileges or run the command as a root to enable the required ports.
# Syslog (udp)
Configuration of log shipper 'syslog (udp)'.
This section explains how you can configure 'syslog' like a log shipper.
In order for syslog component to send the log details to the event gateway, users have to configure two elements.
1. Event Gateway Endpoint
2. Systems syslog configuration on VMWare vSphere environment
**Step 1:**
An example Event Gateway Endpoint configuration is captured in the below configuration snippet.
```
// Gateway Endpoint:
endpoints:
- name: syslog_udp_events
enabled: true
type: syslog_udp
port: 514
attrs:
site_code: cfx_dc1
archive_name: network_syslogs
stream: syslog-udp-stream-01
```
**Step 2:**
An example VMWare vSphere configuration is captured in the below configuration snippet.
From vCenter Configuration,
Select host --> Configuration --> Advanced Settings -->Syslog-->Syslog.global.loghost
Provide target event gateway udo details --> udp\://eventgatewayip:port
Example: **udp\://\:514**
Note: User is expected to have sufficient privileges to enable/update vCenter configuration
# Splunk forwarder (Windows and Linux)
Configuration of log shipper 'splunk forwarder'
This section explains how you can configure 'Splunk' like a log shipper.
In order for the Splunk component to send the log details to the event gateway, users have to configure two elements.
1. Event Gateway Endpoint
2. Splunk configuration on Linux/Windows configuration
**Step 1:**
An example Event Gateway Endpoint configuration is captured in the below configuration snippet.
```
Gateway Endpoint:
endpoints:
- name: winodows_events
enabled: true
type: tcp_json
port: 9997
attrs:
site_code: dc1
archive_name: splunk_events
stream: windows-splunk-stream
```
**Step 2:**
Update the input and output.conf file from the below path:
**Input conf file:**
```
cd /opt/splunkforwarder/etc/system/local/
##Example adding messages##
[monitor:///var/log/messages*]
_TCP_ROUTING = *
disabled = false
```
**outputs.conf file:**
```
[tcpout]
defaultGroup = default-autolb-group
[tcpout-server://:9997]
[tcpout:default-autolb-group]
disabled = false
sendCookedData = false
server = :9997
#Example
[tcpout]
defaultGroup = default-autolb-group
[tcpout-server://10.95.131.101:9997]
[tcpout:default-autolb-group]
disabled = false
sendCookedData = false
server = 10.95.131.101:9997
```
**Step 3:**
Restart splunk service
```
cd opt/splunkforwarder/bin
./splunk stop
./splunk start
```
# Winlogbeat (Windows)
Configuration of log shipper 'Winlogbeat'
This section explains how you can configure 'Winlogbeat' like a log shipper.
In order for winlogbeat component from the Windows environment to send the log details to the event gateway, users have to configure two elements.
1. Event Gateway Endpoint
2. Winlogbeat configuration
**Step 1:**
An example Event Gateway Endpoint configuration is captured in the below configuration snippet
```
Gateway Endpoint:
endpoints:
- name: winlog_01 # URL is implicit, http://ip:port/winlogs
type: filebeat
stream: winlogbeat
ssl: false
enabled: true
xpack_features: min
attrs:
site_code: win-dc
port: 8508
```
An example windows based winlogbeat configuration is captured in the below configuration snippet.
Update winlogbeat.yaml file with elastic search details as event gateway IP address:
**Step 2:**
```
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["http://eventgatewayip:8508/winlogs"]
```
```
#Example configuration
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["http://10.95.122.185:8508/winlogs"]
```
**Step 3:**
Restart winlogbeat services from Powershell from the winlogbeat installed path
```
stop-service winlogbeat
start-service winlogbeat
```
# RDA Log Archives
Log Archive Repository
A named archive within a Log Archive Repository. Each repository will contain one more or more archives to store log data.
### Terminology
### Log Archive Storage in Object Storage
The following screen capture explains the structure of log archives storage under s3 compatible object storage.
### CLI Interface
The following sections explain various CLI commands that can be used in the context of RDA Log Archive functionality.
#### **CLI Commands**
#### **CLI Repositories**
### UI in SaaS Portal
Users can access the RDA Log Archive user interface via the SaaS portal.
Login to SaaS Portal UI --> Configuration → Data
### Bots
RDA provides various system-enabled bots in order to use RDA Log Archive functionality. The following captures a few of the bots that are part of RDA.
#### Replay Bot
#### **Save Bot**
### Event Gateway Configuration
Users can enable RDA Log Archive configuration via the RDA event gateway main configuration. Users must enable the configuration for an endpoint to take effect the log archive functionality as shown below.
Note: Refer to the [Event Gateway section](/rda/installation/event-gateway) for more details on Event Gateway
### RDA Log Archive - RDA Client CLI Commands
#### secret-add
CLI to add S3 Compatible secret key credentials
```
[macaw@localhost~]$ rdac secret-add --type logarchive_repo
Configure Integration: S3 Compatible Object Storage for Log Archiving
Name*: deleteme
Host or Endpoint*: 10.95.131.103:9000
Use HTTPS [True]: false
Access Key*: fa337ca7887345b6971b4c4e33f00056rdauser
Secret Key*:
Bucket*: tenants.904a6613553c4e7cab45fe971b4a66bd
Object Prefix* [/]: log_archivals/
Manage Lifecycle of Archives: 2
Delete Old Data After (days) [30]: 7
[macaw@localhost~]$
```
#### secret-list
CLI to show the list of added secrets to the environment
```
[macaw@localhost]$ rdac secret-list
name type saved_time checksum
-- ------------- --------------- -------------------------- --------------------------------
0 longivity-131 logarchive_repo 2022-02-02T11:54:37.992400 0606adcd94b9538e988276d234b03c1a
1 deleteme logarchive_repo 2022-02-02T11:32:18.473626 08231998fea34052c59319ebf636148a
```
#### logarchive-repos
CLI to show the list of logarchive-repos added to the environment.
```
[macaw@localhost]$ rdac logarchive-repos
+-------------------+--------------------+------------------------------------------+-----------------+
| Repository Name | Endpoint | Bucket Name | Object Prefix |
|-------------------+--------------------+------------------------------------------+-----------------|
| longivity-131 | 10.95.131.103:9000 | tenants.904a6613553c4e7cab45fe971b4a66bd | log_archivals/ |
| deleteme | 10.95.131.103:9000 | tenants.904a6613553c4e7cab45fe971b4a66bd | log_archivals/ |
+-------------------+--------------------+------------------------------------------+-----------------+
[localhost]$
```
#### logarchive-names
CLI to show the list of logarchive-names added to logarchive-repo's (repo name is needed to list the configured names within that repo)
```
[macaw@localhost]$ rdac logarchive-repos
+-------------------+--------------------+------------------------------------------+-----------------+
| Repository Name | Endpoint | Bucket Name | Object Prefix |
|-------------------+--------------------+------------------------------------------+-----------------|
| longivity-131 | 10.95.131.103:9000 | tenants.904a6613553c4e7cab45fe971b4a66bd | log_archivals/ |
| deleteme | 10.95.131.103:9000 | tenants.904a6613553c4e7cab45fe971b4a66bd | log_archivals/ |
+-------------------+--------------------+------------------------------------------+-----------------+
[macaw@localhost]$
```
#### logarchive-data-read
CLI to show data read operation using logarchive-data-read command
```
[macaw@localhost]$rdac logarchive-data-read --repo longivity-131 --name longivity --speed 1
```
#### Additional CLIs to perform actions on data
rdac l**ogarchive-data-read** --repo longivity-131 --name longivity --speed 2.0 --max\_rows 100
rdac **logarchive-download** --repo longivity-131 --name longivity --flatten --out /home/macaw/deleteme
rdac **logarchive-data-size** --repo longivity-131 --name longivity
```
[macaw@localhost]$rdac logarchive-data-read --repo longivity-131 --name longivity --speed 2.0 --max_rows 100
[macaw@localhost]$rdac logarchive-download --repo longivity-131 --name longivity --flatten --out /home/macaw/deleteme
[macaw@localhost]$rdac logarchive-data-size --repo longivity-131 --name longivity
```
# RDA - Administration
RDA - Administration
### Change the Default Password
The RDA software install configures the default username/login password as 'rdademo/rdademo1234'. Please follow the below steps to change it.
**Step 1:** Stop the RDA services (Note: Please make sure to change the directory to the RDA install folder, the below examples are for reference only.)
**Windows:**
```
cd c:\rda-docker-compose-with-ssl
```
**Linux / Mac OS:**
```
cd /home/macaw/rda-docker-compose-with-ssl
```
**Step 2:** Edit **docker-compose.yml** file using your favorite editor and update the default password with a new one, save the file.
**Step 3:** Run the below command to start the **RDA** service.
```
docker-compose -f docker-compose.yml up -d
```
**Step 4:** Login to RDA UI using a browser (enter the URL as **https\://\:9998)** and login with the new password.
### Start and Stop RDA Service:
Note: Please make sure to change the directory to RDA install folder, below examples are for reference only.
**Windows:**
```
cd c:\rda-docker-compose-with-ssl
```
**Linux / Mac OS:**
```
cd /home/macaw/rda-docker-compose-with-ssl
```
To stop the **RDA** service
**Windows:**
```
cd c:\rda-docker-compose-with-ssl
docker-compose stop
```
**Linux / Mac OS:**
```
cd /home/macaw/rda-docker-compose-with-ssl/
docker-compose stop
```
To start the **RDA** service
**Windows:**
```
cd c:\rda-docker-compose-with-ssl
docker-compose stop
```
**Linux / Mac OS**
```
cd /home/macaw/rda-docker-compose-with-ssl/
docker-compose start
```
###
### Check RDA service status:
# RDA - Configuration
**RDA** service reads and loads the configuration from config.yml file. It's location is under \~/cfx/cfxdx/config directory. As part of the installation, it comes with some default configuration settings defined within the config.yml file.
Below are the configuration settings within the config.yml file of RDA service.
```
multisource: yes
platform_config:
minio_config:
host: :9000
secure: false
access_key:
secret_key:
options:
data_saver_type: minio
data_saver_local_path: /tmp/cfxdm_saved_data/
data_saver_minio_bucket_name: cfxdx-data
data_saver_minio_object_prefix: cfxdm-saved-data/
# Data Extensions or Datasource integrations
sources:
# ServiceNow extension. Integration for Tickets, Change Requests and CMDB data
- name: snow
type: servicenow
instance: ""
username: ""
password: ""
$secure: [ "instance", "username", "password" ]
tags:
- tag: incidents
table: incident
label: ServiceNow Incidents
update-mode: append
fields:
mandatory: [ "short_description", "description"]
optional: []
- tag: incidents-update
table: incident
label: ServiceNow Incidents
update-mode: update
fields:
ids: [ "number" ]
data: [ "short_description", "description" ]
- tag: change-requests
table: change_request
label: ServiceNow Change Requests
- tag: cmdb-config-items
table: cmdb_ci
label: ServiceNow CMDB All Configuration Items
- tag: cmdb-computers
table: cmdb_ci_computer
label: ServiceNow CMDB Computers
- tag: cmdb-network
table: cmdb_ci_netgear
label: ServiceNow CMDB Network
update-mode: append
fields:
mandatory: [ "name", "model_id", "ip_address" ]
# SQLite extension.
- name: mylocaldb
type: sqlite
dbpath: '/tmp/output/mydb.db'
tags:
- tag: table1
table: table1
update-mode: append
- tag: table2
table: table2
update-mode: replace
# Local file extension. To explore and visualize the data from CSV files.
- name: localfiles
type: file
# CloudFabrix Machine Learning (ML) extension for Clustering & Prediction.
- type: cfxai_clustering
name: cfxusml
tags:
- tag: logclustering
type: cluster
cluster_columns: [ "description" ]
pickle_path: /tmp/output/models/clustering/
minimum_cluster_size: 50
minimum_sample_size: 1
- tag: logprediction
type: predict
cluster_columns: [ "description" ]
pickle_path: /tmp/output/models/clustering/
# CloudFabrix Machine Learning (ML) extension for Regression analysis.
- type: cfxai_regression
name: cfxml
tags:
- tag: '1hour'
frequency: '1H'
timestamp-column: timestamp
timestamp-format: 'ms'
value-column: value
# CloudFabrix Machine Learning (ML) extension for Classification.
- name: cfxsml
type: cfxai_classification
tags:
- tag: classification-working
type: classify
input_nlp_columns: [ "Summary" ]
target_column: "Issue Type"
pickle_path: /tmp/output/models/issues/
- tag: classificationprediction
type: predict
pickle_path: /tmp/output/models/issues/classifier.pickle
```
### Configuration parameter details:
| **Parameter Name** | **Parameter Description** |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| multisource | Global parameter. Valid values are '**yes**' (default) or '**no**'. When multiple extensions or datasources need to be integrated, set it to '**yes**'. It is a mandatory parameter. |
| sources | Global parameter. Allows to define one or more extensions or datasources for integration. It is a mandatory parameter. |
# RDA - User Guide
{% hint style="info" %}
You can find 'How to use RDA notebooks' as part of your AIOps Studio home.
{% endhint %}
{% hint style="warning" %}
Please note that both tags and bots are interchangeably used.
{% endhint %}
### Built-in RDA functions
The following are the most commonly used built-in functions
| Function (Python) | Terminal | Description |
| -------------------------------- | -------- | ------------------------------------------------------------------------- |
| bots() | bots | View / Search list of bots from all sources |
| plugins() | plugins | View / Search list of installed plugins. |
| api\_models() | | View / Search API Models for the bots that start with '@' (API Endpoints) |
| toolkit() | | a tabbed group of reports with Bots, API Models, Plugins, and Datasets |
| check\_connectivity() | | Check connectivity to data sources and verify credentials if applicable |
| pexec(str) | | Execute pipeline and return the resulting dataframe. |
| viz(df) | | Visualize the dataframe |
| add\_dataset(data or file, name) | | Add dataframe, file or URL as dataset in the RDA dataset repository |
| get\_dataset(name) | | Get previously saved dataset |
| list\_datasets() | | Show list of all saved datasets |
| | | |
### Viewing Datasources and Tags
### plugins()
View / Search all installed RDA extensions.
### check\_connectivity()
Verify network access to all configured datasources. If their credentials are used, verify the credentials also. This function also takes an optional argument which can be a datasource name (ex: `"snow"`)
### tags()
View / Search all datasource, datasink, and control bots
### api\_models()
This function displays a list of all bots that are configured as API Endpoints. These bots are typically prefixed with an @ symbol. For each selected bot, this will display list parameters that should be provided as a combination of *and* (*&*) statements.
### toolkit()
This function will display Bots, Plugins, API Models and Datasets
### list\_datasets()
This method is identical to bot **@dm:recall** *name = "dataset-name"*
# RDA - AIOps Studio
CloudFabrix'x AIOps Studio
### AIOps Studio:
**AIOps Studio** allows users to access it using any browser for data exploration, building and validating data pipelines, solution packages, etc. from simple to complex use-cases.
You can access **RDA** from browser using **https\://\:9998**
{% hint style="info" %}
The default username is rdademo and the default password is rdademo1234
Please refer [**RDA - Installation**](https://app.gitbook.com/@cloudfabrix/s/docs/rda/installation) section under Windows OS /Linux OS for more information.
{% endhint %}
**Step 1:** Enter login password to login into **RDA UI**
*Note: Respective browsers (Chrome, Firefox, Edge) will prompt users to provide a username /password. Please make a note to clear cache before ccessing RDA UI*
**Step 2:** Accessing Terminal from main landing Page
**Step 3:** Accessing AIOps Studio
**Step 4:** AIOps Studio Landing Page
**Step 5 :** Display available AIOps Studio Functions
**Step 6 :** Explore Data Automation bots
# AIOps Studio - Solution Packages
Solution packages provides a packaged solution that includes user credentials, pipelines.
This AIOps studio option provides, users to import a solution package that is pre-packaged for automation and other solutions. It comprises pipelines in JSON files along with datasets in CSV file formats packaged in standard zip file format.
AIOps allows users to import solution packages and allows users to pick and choose pipelines as per need.
#### Solution Package - Import
File Path or URL: File can be uploaded to AIOps Studio (Jupyter folder location via UI) or Downloaded from a URL
Password for decrypting configurations: Password that was used while exporting the solution packages (can be empty in case if there is no password used during export).
#### Solution Package - Export
Name - Name of the solution package that the user is creating.
Version - Version a user intends to attach for versioning solution packages.
Description - Description of the solution package.
Pipelines - User-created /developed pipelines that are packaged (intended to package as part of solution package).
Datasets - User-created datasets that are packaged (intended to package as part of solution package)
Configurations - Datasource configurations that are packaged (intended to package as part of solution package).
Password for encryption configurations - Password to protect solution package while exporting (a user can leave this field empty)
Export To File - To export a file in zip-file format.
# AIOps Studio - Pipelines
AIOps Studio pipeline functionality framework that enables users to automate end-to-end tasks.
#### Pipeline - Adding a New Pipeline
#### Pipeline - Editing a Pipeline
#### Pipeline - Delete a Pipeline
# AIOps Studio - Explore
This section provides details on exploring AIOps Studio functionality.
#### Explore - Data Automation Bots
AIOps Studio provides extensive set of out-of-the box data-automation bots to end-users to automate various data exchange/mapping/filtering capability.
#### Explore - API Models for Data Automation Bots
#### Explore - Datasets
This section provides visibility into user defined datasets.
# AIOps Studio - Administration
AIOps Studio Administration functionality
#### Manage Bot Sources (Add/Edit/Delete)
Users will be be able to add external bot sources using AIOps Studio UI as shown below:
#### Check Connectivity for all Configured Sources
This option provides quick validation (test connectivity) from RDA /AIOps Studio to configured target sources (e.g. VMware, VRops, Elasticsearch, etc.)
#### View all the plugins available via Plugins
# RDA CLI in UI
RDA CLI in UI Terminal
**RDA UI** provides a built-in terminal which allows user to access RDA CLI.
{% hint style="warning" %}
Please note that both tags and bots are interchangeably used.
{% endhint %}
After entering the RDA CLI terminal, enter '**help**' command to see all available commands and their usage.
Below are the usage details for each command.
**browse:** Shows the last data frame into the default browser. It can be invoked using the short command 'br'
**cap**: List all bots and their declared capabilities.
* **get-data:** It can read and fetch the data from the selected bot
* **data-update:** It can update (write-back) the data into the selected bot
* **data-stream:** It can stream the data in chunks from the select bot until the data exhaustion. It is primarily used in ML (Machine Learning) use cases.
* **count:** It allows to query the data and returns the count of matched rows from the query.
Run '**cap**' command and specify extension type to view it's specific available bots and their **capabilities**. (ex: cap vrops / appdynamics / vmware-vcenter etc..)
**check:** It checks and verifies network access and the provided credentials for all or selected extensions or data-sources.
Run '**check**' commands without any arguments verifies access for all extensions or data-sources.
{% hint style="info" %}
For some extension types (like file, control, etc..), the status shows as '**Does not use any credentials**' which means, these extensions do not require any credentials & network details to access the data from them.
{% endhint %}
**conf:** It shows the current configuration of the selected bot. Before running this command, you need to select one of the available bot.
**limit**: Limit defines max no of records (or rows) to be retrieved while querying the data. By default it is set to 1000. To remove the limit of 1000, set it to 0. It means, it will retrieve all of the available records from the selected bot.
{% hint style="warning" %}
Caution: Data query may take minutes to hours when the selected bot has thousand or millions of records when the limit is set to 0. Additionally, limit the query scope by applying filters from the source. For basic data exploration, leave the limit setting with defaults or adjust it accordingly.
{% endhint %}
**page:** Limits or set the page size (retrieve no of records / rows per page) for paginated queries. The default value is set to 50.
**plugins:** Lists all of the available and loaded plugins with in the **RDA** service**.** It means the listed plugin is supported as an extension and available to use.
#### List of Available Plugins
| S.No | Plugin | Module |
| ---- | --------------------- | ------------------------------------------------------------------------------------------- |
| 1 | aiaexpress | cfxdx\_ext\_aiaexpress.aiaexpress\_source.AiaexpressSource |
| 2 | appdynamics | cfxdx\_ext\_appdynamics.appdynamics\_source.AppdynamicsSource |
| 3 | aws | cfxdx\_ext\_aws.aws\_source.AWS\_Source |
| 4 | aws-cloudwatch | cfxdx\_ext\_aws\_cloudwatch.awscw\_source.AWSCW\_Source |
| 5 | azure | cfxdx\_ext\_azure.azure\_source.AZURE\_Source |
| 6 | azure-insights | cfxdx\_ext\_azure\_insights.azurein\_source.AZUREIN\_Source |
| 7 | cfxai\_classification | cfxdx\_ext\_cfxai\_classification.cfxai\_classification\_source.Cfxai\_ClassificationSource |
| 8 | cfxai\_regression | cfxdx\_ext\_cfxai\_regression.cfxai\_regression\_source.CFXAI\_RegressionSource |
| 9 | cfxdm | cfxdx\_ext\_cfxdm.cfxdm\_source.CfxdmSource |
| 10 | cisco-support | cfxdx\_ext\_cisco\_support.cisco\_source.CiscoSource |
| 11 | cloud\_defense | cfxdx\_ext\_cloud\_defense.cloud\_defense\_source.CloudDefense\_Source |
| 12 | consul | cfxdx\_ext\_consul.consul\_source.Consul\_Source |
| 13 | control | cfxdx.controls.ctrl\_tags\_source.ControlTagSource |
| 14 | datagen | cfxdx\_ext\_datagen.datagen\_source.DatagenSource |
| 15 | datanetwork | cfxdx\_ext\_datanetwork.datanetwork\_source.DatanetworkSource |
| 16 | dynatrace | cfxdx\_ext\_dynatrace.dynatrace\_source.DynatraceSource |
| 17 | elasticsearch | cfxdx\_ext\_elasticsearch.es\_source.ES\_Source |
| 18 | email\_reader | cfxdx\_ext\_email.email\_source.ReadEmailSource |
| 19 | email\_sender | cfxdx\_ext\_email.email\_source.EmailSource |
| 20 | file | cfxdx\_ext\_file.csvfile\_source.FileSource |
| 21 | hdfs | cfxdx\_ext\_hdfs.hdfs\_source.HDFS\_Source |
| 22 | ibm\_watson | cfxdx\_ext\_ibm\_watson.ibm\_watson\_source.IBM\_Watson\_Source |
| 23 | istio | cfxdx\_ext\_istio.kiali\_source.Kiali\_Source |
| 24 | jira | cfxdx\_ext\_jira.jira\_source.JIRA\_Source |
| 25 | kafka | cfxdx\_ext\_kafka.kafka\_source.Kafka\_Source |
| 26 | meraki | cfxdx\_ext\_meraki.meraki\_source.Meraki\_Source |
| 27 | mysql | cfxdx\_ext\_mysql.mysql\_source.MysqlSource |
| 28 | nagios | cfxdx\_ext\_nagios.nagios\_source.NagiosSource |
| 29 | openai | cfxdx\_ext\_openai.openai\_source.OpenAI\_Source |
| 30 | oracle | cfxdx\_ext\_oracle.oracle\_source.OracleSource |
| 31 | prometheus | cfxdx\_ext\_prometheus.prom\_source.PrometheusSource |
| 32 | prtg | cfxdx\_ext\_prtg.prtg\_source.PrtgSource |
| 33 | restclient | cfxdx\_ext\_restclient.rest\_source.RESTClientSource |
| 34 | rubrik | cfxdx\_ext\_rubrik.rubrik\_source.RubrikSource |
| 35 | servicenow | cfxdx\_ext\_servicenow\.snow\_source.ServiceNow\_Source |
| 36 | slack | cfxdx\_ext\_slack.slack\_source.Slack\_Source |
| 37 | splunk | cfxdx\_ext\_splunk.splunk\_source.Splunk\_Source |
| 38 | sqlite | cfxdx\_ext\_sqlite.sqlite\_source.SqliteSource |
| 39 | teamcity | cfxdx\_ext\_teamcity.teamcity\_source.TeamCity\_Source |
| 40 | vmware-vcenter | cfxdx\_ext\_vmware.vmware\_source.VMWare\_Source |
| 41 | vrops | cfxdx\_ext\_vrops.vrops\_source.VROps\_Source |
| 42 | webhook | cfxdx\_ext\_webhook.webhook\_source.WebhookSource |
{% hint style="info" %}
You can check for available plugins in your environment by running following command at terminal.
\> plugins
A sample screen is as seen below.
{% endhint %}
**source:** Lists all of the available and configured extensions / datasources along with their bots and capabilities.
**bots:** To list all of the available bots from different extensions. To show available bots for a specific extension / datasource, enter '**bots \**'
**bot:** To switch to a specified '**bot**'. Run '**bots**' command to see all of the available bots.
**sink:** Lists all of the available bots that supports data-update capability (write the data) from a different datasource.
**meta:** Queries and lists available columns (schema) within the selected bot. Select a bot before running this command.
{% hint style="info" %}
**meta** command is not supported for all bots. For unsupported tags, to view the available columns, set the **limit** to 10, run '**data**' (or supported query within the bot) and press '**c**' within the table.
{% endhint %}
**viz:** It allows user to visualize the data after retrieving the data using a query from the selected bot. By default, after the data query, it opens data visualization in tabular form automatically. However, when you quit (using '**q**' key) from the visualization table, in order to visualize the data again, you can execute '**viz**' command to get back into the visual tabular view.
{% hint style="info" %}
**Note:** After executing the query through 'data' which retrieves the data for the selected bot, it caches the retrieved data in memory until the current bot is switched to another bot or 'data' query is executed again with in the selected bot.
{% endhint %}
**pipe:** It is primarily is used to manage data pipeline configuration and executions.
**eval:** It evaluates the specified command which allows the user to use and run python functions/expressions within the dataframe management commands.
# AIOps Studio - Examples
This section provides various examples and use-cases using AIOps Studio
{% content-ref url="/pages/-MYwE\_Lw000z4oQSTq6O" %}
[File Operations](/rda/rda-userguide/rda-aiops-studio/examples-jupyter/file-related-operations)
{% endcontent-ref %}
{% content-ref url="/pages/-MZErf42EOrAc80gTjhS" %}
[Data Management Operations - cfxdm](/rda/rda-userguide/rda-aiops-studio/examples-jupyter/cfxdm-operations)
{% endcontent-ref %}
{% content-ref url="/pages/-MZEwbRoQmsNV9JT56s\_" %}
[Metadata - cfxdm-dm:metadata](/rda/rda-userguide/rda-aiops-studio/examples-jupyter/cfxdm-metadata)
{% endcontent-ref %}
{% content-ref url="/pages/-M\_2qJdta4htfzvzrSju" %}
[Filters - cfxdm - dm:filter](/rda/rda-userguide/rda-aiops-studio/examples-jupyter/cfxdm-dm-filter)
{% endcontent-ref %}
# File Operations
This section covers how AIOps Studio can be used for perform file based operations
## Loadfile - Loading files into AIOps Studio
This section explains how users can upload/load a file (in CSV format) into AIOps Studio. To simplify operations, two example files are provided for users to upload to the local RDA/AIOps environment.
Example CSV files are available from the following links.
1. [employees.csv ](https://macaw-amer.s3.amazonaws.com/rda/data/employees.csv) -- This file contains a set of Employee details.
2. [hosts.csv ](https://macaw-amer.s3.amazonaws.com/rda/data/hosts.csv)-- This file contains a set of IP address/Host details.
**Step 1**: Download the above two files to your local file system using a standard web browser.
**Step 2**: Upload (load) the above two into your RDA system as shown below.
* Using a web browser, connect to the RDA system \:9998>
* Click on the CFXDX Python 3 icon from the main landing page of the RDA launcher (as shown below)
* Launch AIOps studio using the RDA provided text field to start AIOps studio.
* Upload the two files using the file browser upload button provided as part of RDA UI as shown below:
* Your browser will pop up a file browser window (based on your working environment) to select the downloaded files employees.csv and hosts.csv as shown below.
* Select both the files and upload both the files into your RDA environment. After you have completed uploading files, you will see two files under the file browser of the RDA environment as shown below.
* Now you are ready to load the above two files into AIOps Studio as explained in the following steps.
**Step 3**: Add a new empty pipeline with the name "load-employee-file" as shown below and click the "Save" button (this step will create an empty pipeline and saves it to AIOps studio).
**Step 4**: Add the following pipeline commands into the empty pipeline text field that you have created in the above Step 3.
You can copy the below code into your pipeline and execute that in your environment.\
*`##### This pipeline loads employees.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from employees.csv file`*\
*`##### into local dataset named 'employee-dataset'.`*\
*`##### prints the data that was stored`*\
*`@files:loadfile filename = "employees.csv"`*\
*`--> @dm:save name = ' employee-dataset'`*\
*`--> *dm:filter *`*
**Step 5**: Click the button 'Verify' and Click the 'Save' button (AIOps studio verifies the pipeline syntax and stores the pipeline into the earlier created load-employee-file pipeline as shown below.
**Step 6**: Users can now execute the above-created pipeline by clicking the 'Execute' button. Once the users click the 'Execute' button, AIOPs studio executes the pipeline that will load the user-defined file, save the loaded file into a local dataset named 'employee-dataset' as shown below.
**Step 7**: Verify the data using AIOps Studio using 'Inspect --> Data --> Get Data'.
**Step 8**: Try the above steps with the hosts.csv file to store/display the data loaded by AIOps studio using file load operation.
*`##### This pipeline loads 'hosts.csv' file int AIOps Studio`* \
*`##### AIOps studio stores the data loaded from hosts.csv file`*\
*`##### into local dataset named 'hosts-dataset'.`*\
*`##### prints the data that was stored`*\
*`@files:loadfile filename = "hosts.csv"`*\
*`--> @dm:save name = ' hosts-dataset'`*\
*`--> *dm:filter *`*
## Savefile - Saving data/dataset output to a file from AIOps Studio
This section explains how users can save the data/dataset output from a pipeline into a file (in CSV format) via AIOps Studio. To simplify operations, two examples are provided for users to save to the local filesystem using AIOps studio.
Example CSV files are available from the following links.
1. [car-model-to-price.csv](https://macaw-amer.s3.amazonaws.com/rda/data/car-model-to-price.csv) - This file contains a file that was saved from the data output of pipeline execution saved to a file.
2. [device\_details.csv](https://macaw-amer.s3.amazonaws.com/rda/data/device_details.csv) -- This file contains a set of device details saved from the data output of pipeline execution saved as a file.
**Step 1**: Add a new empty pipeline with the name "car-model-pipeline" as shown below and click the "Save" button (this step will create an empty pipeline and saves it to AIOps studio).
**Step 2**: Add the following car-model-to-price pipeline as shown below using the AIOps pipeline.
You can copy the below lines of code into your pipeline and execute that in your environment.
*`##### This pipeline creates temporary car models to price dataset`* \
*`##### and saves the temporarily created dataset into car-models.csv`*\
*`##### File will be saved under ~/cfx/cfxdx/home/nodebooks directory.`* \
*`@dm:empty`*\
*`--> @dm:addrow car_model = 'Honda Civic' & price ='16000' & id = 'hc1'`*\
*`--> @dm:addrow car_model = 'Toyota Corolla' & price ='13000' & id = 'tc2'`*\
*`--> @dm:addrow car_model = 'Ford Focus' & price ='11000' & id = 'ff3'`*\
*`--> @dm:addrow car_model = 'Audi Q5' & price ='49000' & id = 'aq5'`*\
*`--> @files:savefile filename = 'car-model-to-price.csv'`*
Once the above code is added to the empty pipeline that was created via AIOps studio, click the 'Verify' button to verify the pipeline syntax is valid using the studio.
**Step 3**: Execute the pipeline from AIOps studio. AIOps studio executes the pipeline and prints the output as shown below.
The file saved to notebooks directory and file path as shown.
**Step 4**: Execute another pipeline from AIOps studio using the following. AIOps studio executes \
the pipeline and prints the output as shown below.
*`##### This pipeline creates network device details dataset`* \
*`##### and saves the temporarily created dataset into device_details.csv`*\
*`##### File will be saved under ~/cfx/cfxdx/home/nodebooks directory.`* \
*`@dm:empty`*\
*`--> @dm:addrow fqdn = 'itm1.cfx.com' & ip_addr ='192.168.1.1' & id = '1'`*\
*`--> @dm:addrow fqdn = 'dev1.cfx.com' & ip_addr ='192.168.10.1' & id = '2'`*\
*`--> @dm:addrow fqdn = 'qa1.cfx.com' & ip_addr ='192.168.2.1' & id = '3'`*\
*`--> @dm:addrow fqdn = 'fw.cfx.com' & ip_addr ='10.95.101.1' & id = '4'`*\
*`--> @files:savefile filename = 'device_details.csv'`*
## Appendfile - Save to file using filename, if a file already exists, append the data.
***Note: Supports only CSV format.***
This section explains how users can append additional data from the dataset output generated from a \
user-defined pipeline into a file (in CSV format) via AIOps Studio. To simplify operations, examples are provided for using the existing example files.
Example CSV files are available from the following links.
1. [append-to-device-details.csv ](https://macaw-amer.s3.amazonaws.com/rda/data/append-to-device-details.csv)-- In this example, we will use a file (append-to-device-details.csv) from the file system that has a set of devices already added and has other related values (e.g. network device details, etc). After, execution of the pipeline, additional/new data is added/appended to the existing file.
**Step 1**: Add a new empty pipeline with the name "append-to-device-details-pipeline" as shown below and click the "Save" button (this step will create an empty pipeline and saves it to AIOps studio)
**Step 2**: Add the following car-model-to-price pipeline as shown below using the AIOps pipeline.
You can copy the below lines of code into your pipeline and execute that in your environment.
*`##### This pipeline creates network device details dataset`* \
*`##### and saves the temporarily created dataset`* \
*`##### into append-to-device-details.csv`*\
*`##### File will be saved under ~/cfx/cfxdx/home/nodebooks directory.`* \
*`@dm:empty`*\
*`--> @dm:addrow fqdn ='dev-ops1.cfx.com' & ip_addr='192.168.1.10' & id = '5'`*\
*`--> @dm:addrow fqdn ='ops1.cfx.com' & ip_addr='192.168.10.14' & id = '6'`*\
*`--> @dm:addrow fqdn ='router11.cfx.com' & ip_addr='10.95.101.12' & id = '7'`*\
*`--> @files:appendfile filename = 'append-to-device-details.csv'`*
**Step 2**: Verify the above lines of code via AIOps studio using the 'Verify' button on the UI as shown below. AIOps studio will verify the syntax of the pipeline code and display 'OK' to proceed to the next step.
**Step 3**: Next step is to execute the pipeline code via AIOps studio by selecting/clicking the 'Execute' button as shown below. Once the studio, executes the pipeline, it will display the successful execution of steps on the screen. In addition, the existing file will have additional data/dataset appended to the existing file as shown in the below screen capture.
**Step 4**: Verify the data is appended to the existing data using your favorite editor as shown below.
### Datasets to excel file - Export specified datasets to an Excel file, each dataset as a different sheet. Returns summary of export.
This section explains how users can create an excel file with multiple tabs having different datasets via AIOps studio.
Example excel file is available from the following link.
1. [router-switch-firewall.xlsx](https://macaw-amer.s3.amazonaws.com/rda/data/router-switch-firewall.xlsx) : This is an example excel file that is generated after executing the example pipeline.
**Step 1**: Add a new empty pipeline with the name "datasets-to-xlsx-pipeline" as shown below and click the "Save" button (this step will create an empty pipeline and saves it to AIOps studio).
**Step 2**: Add the following datasets-to-xlsx-pipeline code as shown below using the AIOps pipeline.
You can copy the below lines of code into your pipeline and execute that in your environment.
*`##### This pipeline creates multiple logical datasets that in turn will be used`* \
*`##### to save to an excel file with multiple tabs (for each dataset)`*\
*`##### into router-switch-firewall.xlsx`*\
*`##### File will be saved under ~/cfx/cfxdx/home/notebooks directory`*\
\
*`##### Router block of elements -- block-1`*\
*`--> @c:new-block`* \
*`--> @dm:empty`* \
*`--> @dm:addrow fqdn ='router1.cfx.com' & ip_addr='192.168.1.1' & id = '1'`* \
*`--> @dm:addrow fqdn ='router2.cfx.com' & ip_addr='192.168.10.1' & id = '2'`* \
*`--> @dm:addrow fqdn ='router3.cfx.com' & ip_addr='192.168.11.1' & id = '3'`* \
*`--> @dm:addrow fqdn ='router4.cfx.com' & ip_addr='192.168.121.1' & id = '4'`* \
*`--> @dm:save name = 'router-dataset'`*\
\
*`##### Switch block of elements -- block-2`*\
*`--> @c:new-block`* \
*`--> @dm:empty`* \
*`--> @dm:addrow fqdn ='switch1.cfx.com' & ip_addr='192.160.1.1' & id = '1'`* \
*`--> @dm:addrow fqdn ='switch2.cfx.com' & ip_addr='192.161.10.1' & id = '2'`* \
*`--> @dm:addrow fqdn ='switch3.cfx.com' & ip_addr='192.162.10.1' & id = '3'`* \
*`--> @dm:addrow fqdn ='switch3.cfx.com' & ip_addr='192.163.10.1' & id = '4'`* \
*`--> @dm:save name = 'switch-dataset'`*\
\
*`##### Firewall block of elements -- block-3`*\
*`--> @c:new-block --> @dm:empty --> @dm:addrow fqdn ='firewall1.cfx.com' & ip_addr='192.170.1.1' & id = '1' --> @dm:addrow fqdn ='firewall2.cfx.com' & ip_addr='192.171.1.1' & id = '2' --> @dm:addrow fqdn ='firewall3.cfx.com' & ip_addr='192.172.1.1' & id = '3' --> @dm:addrow fqdn ='firewall4.cfx.com' & ip_addr='192.173.1.1' & id = '4' --> @dm:save name = 'firewall-dataset'`*\
\
*`##### This block is adding an empty dataset first, followed by`*\
*`##### three logical datasets that were created from earlier above`* \
*`##### pipeline blocks -- block-4`*\
*`--> @c:new-block`* \
*`--> @dm:empty`* \
*`--> @dm:addrow dataset = "router-dataset" & sheet = "Router"`* \
*`--> @dm:addrow dataset = "switch-dataset" & sheet = "Switch"`* \
*`--> @dm:addrow dataset = "firewall-dataset" & sheet = "Firewall"`*\
\
*`###### pipeline block which stores the logical datasets into final output file -- block-5`*\
*`--> @c:new-block`* \
*`--> @files:datasets_to_xlsx filename = "router-switch-firewall.xlsx" & datasets = "router-dataset|switch-dataset|firewall-dataset"`*
The above set of pipeline code blocks are explained below for each step. The following explanation will clarify mu
Router block of elements -- block-1: In this pipeline block, we are trying to create a simple dataset of three rows for 'router' related data (example data per se). \
*@c:new-block*: In the first line, '@c:new-block' is a keyword to tell studio runtime that the user wants to start a logical new block.\
*@dm:empty*: In the second line, '@dm: empty' is a keyword (data mapper functionality) to tell the studio runtime that the user wants to start an empty dataset to start with.\
*@dm:addrow:* The rest of the lines have '@dm:addrow' tells the studio to add data for each new row into the dataset using different values as shown in the above pipeline code.
**Step 3**: Add the above pipeline code into the newly created pipeline in Step-2, verify the pipeline using the 'Verify' button. The studio will verify the syntax and display 'OK' as shown below.
**Step 4**: Once the pipeline code syntax is verified, run the pipeline code by selecting the 'Execute' button. Studio will execute the pipeline as shown below and stores the dataset(s) into excel file under \~/cfx/cfxdx/home/notebooks folder. Output screenshot is shown below for reference:
**Step 5: Now, users will be able to verify the data written into an excel file using the windows excel tool.**
**Note: You can cross-check the generated data using the reference file '**[**router-switch-firewall.xlsx**](https://macaw-amer.s3.amazonaws.com/rda/data/router-switch-firewall.xlsx)**'**
### Dataset to JSON file - Section explains how to store dataset into a file in JSON format.
This section explains how users can create a dataset using CSV file via AIOps studio. In turn, use AIOps RDA functions to store the imported data from CSV file into the dataset to
{% hint style="info" %}
Download the [incidents.csv](https://macaw-amer.s3.amazonaws.com/rda/data/incidents.csv) file to the local machine as shown below using a standard web browser.
{% endhint %}
**Step 1**: Download 'incidents.csv' to AIOps RDA environment as shown below from the local file system.
**Step 2:** Upload the file 'incidents.csv' to AIOps studio using file-browser (as shown below)
**Step 3:** Add a new empty pipeline with the name "load-incidents-file" as shown below and click the "Save" button (this step will create an empty pipeline and saves it to AIOps studio).
**Step 4**: Add the following pipeline commands into the empty pipeline text field that you have created in above Step 3.
You can copy the below code into your pipeline and execute that in your environment.\
*`##### This pipeline loads incidents.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from incidents.csv file`*\
*`##### into local dataset named 'incident-summary'.`*\
*`##### prints the data that was stored`*\
*`@files:loadfile filename = "incidents.csv"`*\
*`--> @dm:save name = 'incidents-summary'`*\
*`--> *dm:filter *`*
**Step 5**: Click the button 'Verify' and Click the 'Save' button (AIOps studio verifies the pipeline syntax and stores the pipeline into the earlier created load-incidents-file pipeline as shown below.
**Step 6**: Users can now execute the above-created pipeline by clicking the 'Execute' button. Once the users click the 'Execute' button, AIOPs studio executes the pipeline that will load the user-defined file, save the loaded file into a local dataset named '`incident-summary`' as shown below.
**Step 7**: Add the following code snipped to the above created pipeline
You can copy the below code into your pipeline, verify and execute that in your environment.\
\
*`##### This pipeline loads incidents.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from incidents.csv file`*\
*`##### into local dataset named 'incident-summary'.`*\
*`##### prints the data that was stored`*\
*`@files:loadfile filename = "incidents.csv"`*\
*`--> @dm:save name = 'incidents-summary'`*\
*`--> *dm:filter *`* \
*`###### This block is adding an empty dataset first, followed`*\
*`###### by restoring the dataset that was created above and next,`* \
*`###### saving the data into 'incidents.json'`* \
*`--> @c:new-block`*\
*`--> @dm:recall name = 'incidents-summary'`*\
*`--> @files:savefile filename = 'incidents.json'`*\
\-
**Step 8**: User can now execute the pipeline using AIOps Studio. Once the pipeline is executed, RDA loads the dataset and stores the data into 'incidents.json' (in JSON format) as shown below.
### File operations next steps
**Now, users will able to use the above reference examples to try with other CSV/data files.**
# Loop Operations
This section covers how looping functionality can be used in RDA pipelines under AIOps Studio
##
{% hint style="info" %}
Looping functionality will work on the datasets which were created by loading files into RDA and/or via logical dataset creation from other datasets.
{% endhint %}
## 1. Timed loop
Step 1: Create an empty **timed\_loop\_example\_1** using AIOps studio as shown in the below screenshot
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.
\
*`###### Pipeline for timed loop: Sleeps 'interval' seconds after each iteration.`*\
*`###### if the max_iterations is specified, loop will stop after max_iterations reached`*\
*`###### each iteration as variable loop_index set (starts at 0 and increments by 1)`*\
\
*`@c:timed-loop interval = '3' & max_iterations = 10`*\
*`--> @dm:empty`*\
*`--> @dm:addrow message = "This is row number ${loop_index}"`*\
*`--> @dm:save name = "temp-df-${loop_index}"`*\
\
*`--> @c:new-block`*\
*`--> @dm:concat names = "temp-df-.*"`*
Step 3**:** Click verify button to verify the pipeline. RDA will verify the pipeline without any errors (as shown below)
Step 4**:** Click execute button to execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
RDA executes a looping-based pipeline and prints the loop details as shown in the above screenshot. During each iteration/loop, the pipeline screen updates the iteration number as shown in the above screenshot.
After iterating through max\_iterations, pipeline data is stored in output and exits.
Step 5: Verify the data after the timed loop operation is completed as shown in the below screenshot (data is printed for 10 iterations in this example).
## 2. Count loop
Step 1: Create an empty **count\_loop\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.
\
*`###### Pipeline for count loop: starts with 'start' number and increment by using`*\
*`###### 'increment' until the value is greater then equal to 'end' variable`*\
*`###### value for the each iteration is set as 'loop_index'`* \
\
*`@c:count-loop start = 0 & end = 10 & increment = 1`*\
*`--> @dm:empty`*\
*`--> @dm:addrow message = "This is row number ${loop_index}"`*\
*`--> @dm:save name = "temp-df-${loop_index}"`*\
*`--> @c:new-block`*\
*`--> @dm:concat names = "temp-df-.*"`*
Step 3**:** Click verify button to verify the pipeline. RDA will verify the pipeline without any errors (as shown below)
Step 4**:** Click execute button to execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
RDA executes a loop-count-based pipeline and prints the loop details as shown in the above screenshot. During each iteration/loop, the pipeline screen updates the iteration number as shown in the above screenshot.
After iterating till 'end', pipeline data is stored in output and exits.
# Data Management Operations - cfxdm
**RDA** provides comprehensive built-in data management and transformation capabilities through **cfxdm extension and bots.**
This section explains how users can use cfxdm capabilities via AIOps Studio. To simplify operations, example files are provided for users to upload to the local RDA/AIOps environment.
Example CSV files are available from the following links.
1. [surveys.csv](https://macaw-amer.s3.amazonaws.com/rda/data/surveys.csv) -- This file contains a set of survey details.
2. [incidents.csv](https://macaw-amer.s3.amazonaws.com/rda/data/incidents.csv) -- This file contains a set of tickets/incident details
We will use the above file to show working examples of data management operations using AIOps Studio RDA functionality.
# Data mapping - cfxdm - dm:eval
### Eval
Given an expression evaluates the expression string. If performed on the data frame row, it evaluates bypassing the row as a dictionary. If performed on a single value, it expects an additional argument 'key' to be used in the expression.
@param expr: The expression to evaluate. \
@param key: An optional 'key' if evaluated on a single value instead of a dictionary.
This function allows users to dynamically evaluate expressions from a string-based input. If a user passes in a string to eval, it evaluates the input string as an expression. RDA eval internally maps to Python language eval.
* Eval is a built-in- function used in python, eval function parses the expression argument and evaluates it as a python expression. **In simple words, the eval function evaluates the “String” like a python expression and returns the result as an integer.**
* Eval function is used in situations or applications that need to evaluate mathematical expressions. Also if the user wants to evaluate the string into code then can use the eval function, because the eval function evaluates the string expression and returns the integer as a result.
### Example 1:
Step 1: Create an empty **eval\_example\_1** using AIOps studio as shown in the below screenshot.
This pipeline provides example usage of eval to remove starting and ending characters from a
start using the **index** mechanism
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a simple as array of elements called vmlist.`*\
\
*`##### RDA function eval is used to demo this example.`*\
*`##### This function array index pattern to strip or remove starting and`* \
*`##### ending characters and returns the elements using eval`*\
\
*`@dm:empty`* \
*`--> @dm:addrow vmlist = "['vm-1','vm-2','vm-3']"`*\
*`--> @dm:eval vmlist = "vmlist[1:-1]"`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the function eval to use search pattern as mentioned in the pipeline code and strips the characters at index '1' and '-1' (last index from right), return the string. Eval in the current pipeline uses the underlying Python eval function to perform the logic. In this example, vmlist is an array of vm elements and the pipeline uses the 'eval' function to remove '\[1:-1]' and returns the output string as shown below.
### Example 2:
Step 1: Create an empty **eval\_example\_2** using AIOps studio as shown in the below screenshot.
This pipeline provides example usage of eval to remove starting and ending characters from a string using inbuilt library functions **lstrip, rstrip** that are available.
Step 2: Add the following pipeline code/commands into the above=created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a simple as array of elements called vmlist.`*\
*`##### RDA function eval is used to demo this example.`*\
\
*`##### This pipeline uses lstrip and rstrip library functions`*\
*`##### and rturns the string.`*\
\
*`@dm:empty`* \
*`--> @dm:addrow vmlist = "['vm-1','vm-2','vm-3']"`*\
*`--> @dm:eval vmlist = "vmlist.lstrip('[').rstrip(']')"`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the function eval to use library functions lstrip and rstrip to remove starting char '\[' and ending char ']'. Eval in the current pipeline uses the underlying Python eval function to perform the logic. In this example, vmlist is an array of vm elements and the pipeline uses 'eval' and underlying library functions to strip the characters and returns the output a string as shown below.
### Example 3:
Step 1: Create an empty **eval\_example\_3** using AIOps studio as shown in the below screenshot.
This pipeline provides example usage of eval using expression to substitute string based on the logic.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline adds bunch of rows with ipaddress, hostname and ids`*\
*`##### Some of the elements (ipaddress or hostname), are added empty`* \
*`##### to demo the eval use case`*\
\
*`##### RDA function eval is used to demo this example.`*\
*`##### In this pipeline, eval uses expression to evaluate and substitute`*\
*`##### its value based on whether string is empty or non-empty`*\
*`##### If it is non-empty, value is used or if empty, default string is`* \
*`##### substituted as shown in the output`* \
\
*`@dm:empty`* \
*`--> @dm:addrow ipaddress = '10.10.1.1' & hostname = 'host-1-1' & id = 'a1'`*\
*`--> @dm:addrow ipaddress = '10.10.1.2' & id = 'a2'`*\
*`--> @dm:addrow ipaddress = '10.10.1.1' & id = 'a3'`*\
*`--> @dm:addrow hostname = 'host-1-1' & id = 'a4'`*\
*`--> @dm:addrow d = 'a4'`*\
*`--> @dm:eval hostname = "'' + str(hostname) if hostname else 'Empty Hostname' "`*\
*`--> @dm:eval ipaddress = "'' + str(ipaddress) if ipaddress else 'Empty IPAddress' "`*\
*`--> @dm:map from = 'hostname,ipaddress' & to = 'IP_or_Hostname' & func = 'any_non_null'`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the function eval along with the expression(s) that are part of the pipeline to evaluate expression based on an empty or non-empty string within the expression and substitute default value in case of an empty string (or else, provide value).\
\
In this example, ipaddress and hostname are evaluated within the expression. Once evaluated, empty and non-empty strings are substituted as per pipeline logic and return the output as a string as shown below.
### Example 4:
Step 1: Create an empty **eval\_math\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline adds bunch of rows with integer columns column1, column2 and ids`*\
*`##### to demo the eval use case with math functions`*\
\
*`##### In this pipeline, eval uses integer conversion followed by math function`* \
*`##### as expression to calculate values and substitute its value based on underlying`* \
*`##### library function (pow)`* \
\
*`@dm:empty`* \
*`--> @dm:addrow column1 = '0' & column2 = '1' & id = 'a1'`*\
*`--> @dm:addrow column1 = '1' & column2 = '2' & id = 'a2'`*\
*`--> @dm:addrow column1 = '2' & column2 = '4' & id = 'a3'`*\
*`--> @dm:addrow column1 = '3' & column2 = '8' & id = 'a4'`*\
*`--> @dm:addrow column1 = '4' & column2 = '16' & id = 'a5'`*\
*`--> @dm:eval column3 = " pow(2,(int(column2))) "`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the function eval along with 'integer' conversion and math function 'pow' to perform the mathematical operation, substitute the resulting values, and returns column3 with new values as shown in the below screenshot.
In the above example, RDA uses 'dm:eval' in conjunction with the underlying mathematical expression (along with integer conversion and math function 'pow')
### Example 5:
Step 1: Create an empty **eval\_math\_example\_2** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline adds bunch of rows with`* \
*`##### id, cluster info, date, budget, actual amount allocated to a dummy org.`*\
*`##### to demo the eval use case with math function`*\
\
*`##### In this pipeline, RDA uses eval function along with integer conversion`* \
*`##### followed by math addition as expression to calculate values. In addition,`* \
*`##### pipeline adds a new column 'variance' to extract values.`*\
*`##### RDA dm:filter is used to selectively pick only required columns and print the`*\
*`#### output.`*\
\
*`@dm:empty`* \
*`--> @dm:addrow id = '0' & cluster = 'clust-a' & date ='2014-01-01' & budget = '11000' & actual = '10000'`*\
*`--> @dm:addrow id = '1' & cluster = 'clust-a' & date ='2014-02-01' & budget = '1200' & actual = '1000'`*\
*`--> @dm:addrow id = '2' & cluster = 'clust-b' & date ='2014-03-01' & budget = '200' & actual = '100'`*\
*`--> @dm:addrow id = '3' & cluster = 'clust-b' & date ='2014-04-01' & budget = '200' & actual = '300'`*\
*`--> @dm:addrow id = '4' & cluster = 'clust-c' & date ='2014-05-01' & budget = '400' & actual = '450'`*\
*`--> @dm:addrow id = '5' & cluster = 'clust-c' & date ='2014-06-01' & budget = '700' & actual = '1000'`*\
*`--> @dm:addrow id = '6' & cluster = 'clust-c' & date ='2014-07-01' & budget = '1200' & actual = '1000'`*\
*`--> @dm:addrow id = '7' & cluster = 'clust-c' & date ='2014-08-01' & budget = '200' & actual = '100'`*\
*`--> @dm:eval variance = " str( int(budget) + int(actual) )"`* \
*`--> *dm:filter * get id, date, cluster, budget, actual, variance`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the function 'dm:eval' along with 'integer' conversion and addition in calculating variance values. This pipeline also uses dm:filter to print selective columns after variance calculation. Output is as shown below screenshot.
### Example 6
Step 1: Create an empty **eval\_date\_example\_1** using AIOps studio as shown in the below screenshot.
This pipeline provides example usage of eval using an expression for the 'date' string.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline adds bunch of rows with integer values to a column name 'Daily'`*\
*`##### Pipeline uses RDA 'to_type' function to convert string values to integer,`* \
*`##### uses dm:eval function and math functions to calculate integer to value`* \
*`##### in milliseconds, finally use 'change-time-format' to convert milliseconsds`* \
*`##### to date string.`*\
\
*`##### In this pipeline, RDA uses eval, to_type, change-time-format functions`* \
*`##### along with integer and milliseconds conversion`* \
*`@dm:empty`*\
*`--> @dm:addrow Daily = '1'`*\
*`--> @dm:addrow Daily = '2'`*\
*`--> @dm:addrow Daily = '3'`*\
*`--> @dm:addrow Daily = '4'`*\
*`--> @dm:addrow Daily = '5'`*\
*`--> @dm:addrow Daily = '6'`*\
*`--> @dm:to_type columns = "Daily" & type = int`*\
*`--> @dm:eval Daily = 'int(time_now_as_ms()) - (24*60*60*1000 * Daily )'`*\
*`--> @dm:change-time-format columns = 'Daily' & from_format = 'ms' & to_format = '%Y-%m-%d'`*
Add the above pipeline code and save the pipeline as shown below.
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the function eval along with other RDA functions 'time\_now\_as\_ms', 'change-time-format' to display 'date' in proper user readable format as shown below.
Similar to the above example, users can try other expressions using other columns.
### Example 7
Step 1: Create an empty **eval\_wwpn\_example\_1** using AIOps studio as shown in the below screenshot.
This pipeline provides example usage of eval to properly format WWPN that is stored as a string. This example is especially useful for network/storage element manipulation that uses WWPN.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`#####` This pipeline creates a WWWPN string WWPN "c050760aa8ac0070*"\
*`#####` RDA function eval is used to demo this example*\
*`#####` Below code snippet converts above WWPN to standard format : c0:50:76:0a:a8:ac:00:7*0\
\
*`@dm:empty`*\
*`--> @dm:addrow WWPN = "c050760aa8ac0070"`* \
*`--> @dm:eval WWPN = "WWPN[0:2] + ':' + WWPN[2:4] + ':' + WWPN[4:6] + ':' + WWPN[6:8] + ':' + WWPN[8:10] + ':' + WWPN[10:12] + ':' + WWPN[12:14] + ':' + WWPN[14:16]"`*
Add the above pipeline code and save the pipeline as shown below.
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the function eval along with other RDA functions to convert WWPN that is fed as a string (WWPN stored as a standard string format) to consumable WWPN format as shown in the below screen capture.
### Example 8
Step 1: Create an empty **eval\_wwn\_to\_wwpn\_wwnn\_example\_1** using AIOps studio as shown in the below screenshot.
This pipeline provides example usage of eval for WWN that is stored as a string and converted into WWPN / WWNN. This example is especially useful for network/storage element manipulation that uses WWPN/WWNN.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.
*`##### This pipeline creates a WWN Example: (When WWNN and`* \
*`##### WWPN reported as single`* \
*`##### value/String) - WWN: 50:06:01:60:C8:E0:06:A8:50:06:01:64:48:E0:06:A8`*\
*`##### RDA function eval is used to demo this example`*\
*`##### Below code snippet converts above`* \
*`##### WWN: 50:06:01:60:C8:E0:06:A8:50:06:01:64:48:E0:06:A8`*\
*`##### Below bots splits the WWN to WWNN and WWPN as`*\
*`##### WWNN: 50:06:01:60:C8:E0:06:A8`* \
*`##### WWPN: 50:06:01:64:48:E0:06:A8`*
*`@dm:empty`* \
*`--> @dm:addrow WWN = "50:06:01:60:C8:E0:06:A8:50:06:01:64:48:E0:06:A8"`* \
*`--> @dm:map attr = 'WWN' & func = 'replace' & oldvalue = ':' & newvalue = '' --> @dm:map from = 'WWN' & to = 'WWNN'`* \
*`--> @dm:map from = 'WWN' & to = 'WWPN'`* \
*`--> @dm:eval WWNN = "WWNN[0:2] + ':' + WWNN[2:4] + ':' + WWNN[4:6] + ':' + WWNN[6:8] + ':' + WWNN[8:10] + ':' + WWNN[10:12] + ':' + WWNN[12:14] + ':' + WWNN[14:16]"`* \
*`--> @dm:eval WWPN = "WWPN[16:18] +':'+ WWPN[18:20] + ':' + WWPN[20:22] + ':' + WWPN[22:24] + ':' + WWPN[24:26] + ':' + WWPN[26:28] + ':' + WWPN[28:30] + ':' + WWPN[30:32]"`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the function eval along with other RDA functions to convert WWN that is fed as a string (WWN stored as a standard string format) to consumable WWNN and WWPN format as shown in the below screen capture.
### Example 9
Step 1: Create an empty **eval\_mac\_example\_1** using AIOps studio as shown in the below screenshot.
This pipeline provides example usage of eval function to convert a MAC address which is in non-standard format to standard format by filling in appropriate 0 char buffer at respective places. MAC is treated as string input in the current example and with a dot (.) as a delimiter
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a MAC Example in a non-standard format`*\
*`##### MAC: 7eb60ae8c202`*\
*`##### RDA uses eval and other function to convert non-standard format`* \
*`##### to standard MAC representation`* \
*`@dm:empty`* \
*`--> @dm:addrow MAC = "7eb60ae8c202"`* \
*`--> @dm:eval MAC = "MAC[0:2]+':'+MAC[2:4] + ':'+MAC[4:6] + ':'+MAC[6:8] + ':'+MAC[8:10] + ':'+MAC[10:12]"`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA converts raw MAC address to appropriate MAC using eval function using RDA pipeline as shown in the below screen capture.
### Example 10
Step 1: Create an empty **eval\_mac\_example\_2** using AIOps studio as shown in the below screenshot.
This pipeline provides example usage of eval function to convert a MAC address which is in a non-standard format and also missing some of the digits that are needed for MAC address representation.\
\
An example MAC is as shown below (in standard string format):
**MAC: 7e.b6.a.e8.c2.2**
\
The below example pipeline captures a mechanism to convert incorrect MAC address correct MAC address format. This example pipeline is useful for any network/storage pipelines where accurate MAC address representation is required
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a MAC Example in a non-standard format`*\
*`##### and missing chars in MAC:` 7e.b6.a.e8.c2.2*\
*`##### RDA uses eval and other function to convert non-standard format`* \
*`##### to standard MAC representation (First, it converts single digit`* \
*`##### to 2 digits by adding 0 (zero) as prefix and 2nd bot`* \
*`##### replaces '.' (dot) with ':' (colon) )`*\
*`@dm:empty`* \
*`--> @dm:addrow MAC = "7e.b6.a.e8.c2.2"`* \
*`--> @dm:eval MAC = '".".join([s.zfill(2) for s in MAC.split(".")])'`* \
*`--> @dm:map attr = 'MAC' & func = 'replace' & oldvalue = '.' & newvalue = ':'`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA converts raw MAC address to appropriate MAC using eval function using RDA pipeline as shown in the below output screen capture.
# Filters - cfxdm - dm:filter
Filtering related cfxdm:filter tag functionality
**dm:filter:** This cfxdm bot allows the user to apply basic simple to complex filters on the retrieved data from a different extension/data source using CFX query language.
This bot is useful when an extension /data source does not support filtering the data at the source using cfx query language. So, the pre-requisite is to retrieve the data before using this (dm:filter) bot.
**dm:filter syntax:**
* **dm:filter *****\***** :** Use CFX query language to apply the filter. If you do not want to apply any query and to include all of the ingested data, use **'\*'** (without any quotes).
* **dm:filter *****\***** get *****COLUMN\_Name\_1, COLUMN\_Name\_2, COLUMN\_Name\_3***** .... :** Use this syntax to limit the scope to only the selected columns. It also maintains the selected column's specified order.
* **dm:filter *****\***** get *****COLUMN\_Name\_1***** as '*****New\_COLUMN\_Name\_1*****', *****COLUMN\_Name\_2***** as '*****New\_COLUMN\_Name\_2*****', *****COLUMN\_Name\_3***** as '*****New\_COLUMN\_Name\_3'*****... :** Use this syntax to limit the scope to only the selected columns and rename them with new column names. It also maintains the selected column's specified order.
{% hint style="info" %}
Please refer [**CFX query language**](/cfxql-cfx-query-language) section for detailed information on supported queries and their usage/syntax with examples.
{% endhint %}
The current example will use "surveys.csv" provided as a dataset to explain the functionality.
**Step 1**: Download the "[surveys.csv](https://macaw-amer.s3.amazonaws.com/rda/data/surveys.csv)" file to the local machine as shown below using a standard web browser.
**Step 2**: Upload (load) the above file into your RDA system as shown below.
* Using a web browser, connect to the RDA system \:9998>
* Click on the CFXDX Python 3 icon from the main landing page of the RDA launcher (as shown below)
* Launch AIOps studio using the RDA provided text field to start AIOps studio.
**Step 3**: Upload the above downloaded CSV file into AIOps studio as shown below.
Select the upload option from the studio environment (as shown below).
Select the file and upload it into the studio environment (as shown below).
The studio will show the uploaded file in the left panel (as shown below).
**Step 4**: Create an empty dataset 'cfxdm-basic-filter-example" pipeline and save that in AIOps studio and get the data from surveys.csv file using \*dm: filter
**Example 1:** \
This example explains RDA 'dm:filter' functionality to filter selective columns/data from the entire dataset.
**Step a:** \
Get the data from surveys.csv using 'dm:filter'. Below shown columns (highlighted) are from the surveys.csv file. The user selects only the following filtered columns instead of a complete set of columns.
* record\_id
* plot\_id
* year
* month
* other columns
{% hint style="info" %}
**Note**\
**-->** symbol represents piping the data between two extension bots, the output of an extension bot becomes an input to another extension bot. It is similar to using the pipe (I) command in Unix/Linux OS.
{% endhint %}
**Step b:** \
Add the following pipeline code/commands into the empty pipeline text field that you have created in the above step.
You can copy the below code into your pipeline and execute that in your environment.\
*`##### This pipeline loads surveys.csv file into AIOps studio stores`* \
*`##### the data loaded from surveys.csv file into local dataset named`* \
*`##### 'cfxdm-basic-filter'. Once the data is saved in 'cfxdm-basic-filter'`* \
*`##### dataset, RDA functionality '*dm:filter' is used filter only few`* \
*`##### names from the complete dataset that was read from csv file.`*\
\
*`@files:loadfile filename = "surveys.csv"`*\
*`--> @dm:save name = 'cfxdm-basic-filter'`*\
*`--> *dm:filter * get record_id,plot_id, month,year`*
**Step c:** \
Verify the pipeline code using the 'Verify' button on AIOPs studio.
**Step d:** \
Execute the pipeline code using the 'Execute' button on AIOPs studio as shown below.
**Step e:** \
Verify the dm:filter functionality is filtering the dataset and prints the selected columns as shown below.
AIOps Studio --> Studio --> Execute --> Inspect --> Data --> Get Data (button)
**Example 2:**\
This example explains RDA 'dm:filter' functionality to filter selective columns/data from the entire dataset. In addition, this showcases how to dynamically rename the existing dataset columns to user defined column names.
**Step a:** \
Get the data from surveys.csv and modify the column names. Below shown columns (highlighted) are from the surveys.csv file. Now, we will use RDA's 'dm:filter' mechanism to select only few columns and rename those into custom column names.
* record\_id
* plot\_id
* year
* month
* other columns
**Step b:** \
Add the following pipeline code/commands into the empty pipeline text field that you have created in the above step.
You can copy the below code into your pipeline and execute that in your environment.\
*`##### This pipeline loads surveys.csv file into AIOps studio stores`* \
*`##### the data loaded from surveys.csv file into local dataset named`* \
*`##### 'cfxdm-filter-with-column-rename'. Once the data is saved in`* \
*`##### 'cfxdm-filter-with-column-rename' dataset, RDA functionality`* \
*`##### '*dm:filter' is used to filter only few columns/names from`* \
*`##### the complete dataset that ##### was read from csv file and rename`* \
*`#### those with user defined names as follows.`*\
\
*`@files:loadfile filename = "surveys.csv"`*\
*`--> @dm:save name = 'cfxdm-basic-filter'`*\
*`--> *dm:filter * get sex as 'Gender',record_id as 'Record_Indentifier', plot_id as 'Plot_Identifier', month as 'Month', year as 'Year'`*
**Step c:** \
Verify the pipeline code using the 'Verify' button on AIOPs studio.
**Step d:** \
Execute the pipeline code using 'Execute' button on AIOPs studio as shown below.
**Step e:** \
Verify the dm:filter functionality is filtering the dataset and prints the selected columns as shown below.
AIOps Studio --> Studio --> Execute --> Inspect --> Data --> Get Data (button)
**Example 3:**
Get the data from 'survey.csv' data file and using CFX query language, filter the data that matches '**Male**' under column '**Gender**' and limit the column's scope only to Year '**1983**'
**Step a:** \
Get the data from surveys.csv and modify the column names using filtering mechanism to get columns with conditional logic using CFXQL query to select data with column 'Gender' is 'M' and limit the year to '1983.
**Step b:** \
Add the following pipeline code/commands into the empty pipeline text field that you have created in the above step.
You can copy the below code into your pipeline and execute that in your environment.\
*`##### This pipeline loads surveys.csv file into AIOps studio stores the`* \
*`#### data loaded from surveys.csv file into local dataset named`* \
*`#### 'cfxdm-filter-with-column-logic'. Once the data is saved in`* \
*`#### 'cfxdm-filter-with-column-logic' dataset, RDA functionality`* \
*`#### '*dm:filter' is used to filter the columns with user defined`* \
*`#### logic (of Gender is 'M'and Year is '1983')`* \
\
*`@files:loadfile filename = "surveys.csv"`*\
*`--> @dm:save name = 'cfxdm-basic-filter'`*\
*`--> *dm:filter * get sex as 'Gender',record_id as 'Record_Indentifier', plot_id as 'Plot_Identifier', month as 'Month', year as 'Year'`*\
*`--> *dm:filter Gender equals 'M' & Year equals '1983'`*
**Step c:** \
Verify the pipeline code using the 'Verify' button on AIOPs studio.
**Step d:** \
Execute the pipeline code using the 'Execute' button on AIOPs studio as shown below.
**Step e:** \
Verify the dm:filter functionality is filtering the dataset and prints the selected columns as shown below.
AIOps Studio --> Studio --> Execute --> Inspect --> Data --> Get Data (button)
Users can download the [incidents.csv](https://macaw-amer.s3.amazonaws.com/rda/data/incidents.csv) file and explore the 'dm:filter' functionality.
Follow the above-mentioned Step 1, Step 2, Step 3 to upload the 'incidents.csv' file to AIOps studio. Once you upload the file into AIOps studio, you will see the file under the left panel of AIOps studio (as shown below).
**Example 4:**
**Step a:** \
Create an empty dataset and load incidents.csv file into AIOps studio
**Step b:** Add the following pipeline code/commands into the empty pipeline text field that you have created in the above step.
You can copy the below code into your pipeline and execute that in your environment.\
*`#### This pipeline loads incidents.csv file into AIOps studio stores the data`* \
*`#### loaded from incidents.csv file into local dataset named 'cfxdm-incidents'.`* \
*`#### Once the data is saved in 'cfxdm-incidents' dataset, RDA functionality`* \
*`#### '*dm:filter' is used filter #### only few names from the complete`* \
*`#### dataset that was read from csv file.`*\
\
*`@files:loadfile filename = "incidents.csv"`*\
*`--> @dm:save name = 'cfxdm-incidents'`*
This is depicted in the following screen capture.
**Step c:** Verify the above pipeline code and execute the pipeline. AIOps studio will verify the pipeline code and execute the pipeline as shown below.
AIOps Studio --> Studio -->Verify
AIOps Studio --> Studio --> Execute
**Step d:** Verify the data from the execution of the pipeline as shown below.
AIOps Studio --> Studio --> Execute --> Inspect --> Data --> Get Data (button)
**Step e**: Users can now access the metadata from 'incidents.csv' to query data/meta-data from the loaded dataset. In addition, can query and manipulate data using dm: filter functionality.
i. Accessing metadata for the above-loaded dataset is shown in the below screen capture.
ii. Following example explains a simple use case that provides the above-explained dataset along with renaming of columns followed by filtering user required data.
Add the following pipeline code/commands into the empty pipeline text field that you have created in the earlier steps to query only two columns using the \*dm:filter option.
You can copy the below code into your pipeline and execute that in your environment.\
*`##### This pipeline loads incidents.csv file into AIOps studio stores`* \
*`##### the data loaded from ##### incidents.csv file into local dataset`* \
*`##### named 'cfxdm-incidents'. Once the data is saved in 'cfxdm-incidents'`* \
*`##### dataset, RDA functionality '*dm:filter' is used filter`* \
*`#### only few names from the complete dataset that was read from csv file.`*\
\
*`@files:loadfile filename = "incidents.csv"`*\
*`--> @dm:save name = 'cfxdm-incidents'`*\
*`--> @dm:map from = 'Incidents ID' & to = 'cfxdm-incidents'`*\
*`--> *dm:filter Status equals 'Resolved' & Source not equals 'Grafana' & Priority = '1 - Critical'`*
Users can copy the above pipeline code and execute to retrieve data for only two columns 'Incident\_ID' that is renamed as 'Ticket' and Summary from the complete incident list. This is shown in the following screenshot.
In addition to the above examples, users are free to explore various ticketing-based queries using the above incidents.csv file and/or using the logically stored datasets within RDA/AIOps studio.
# Data mapping - cfxdm - dm:map
This tag allows the user to manipulate or transform the columns and its values.
**dm:map:** This tag allows the user to manipulate or transform the columns and their values.
Below are some of the operations you can perform using this tag.
* Copy the Column X along with its values as is and create a new Column Y
* Transform values from a Column to something else using 'functions' based on the user's requirement
**dm: map syntax:**
* **dm:map from = '*****COLUMN\_X*****' & to = '*****COLUMN\_Y*****'**
OR
* **dm:map attr = '*****COLUMN\_Y' &***** func *****= "\" & \***
Following two data CSV files are used to explain dm: map functionality
**Step 1**:
Download the "[surveys.csv](https://macaw-amer.s3.amazonaws.com/rda/data/surveys.csv)" file to the local machine as shown below using a standard web browser.\
Download the [incidents.csv](https://macaw-amer.s3.amazonaws.com/rda/data/incidents.csv) file to the local machine as shown below using a standard web browser.
**Step 2**: Upload (load) the above file into your RDA system as shown below.
* Using a web browser, connect to the RDA system \:9998>
* Click on the CFXDX Python 3 icon from the main landing page of the RDA launcher (as shown below)
* Launch AIOps studio using the RDA provided text field to start AIOps studio.
Select the files and upload them into the studio environment (as shown below).
### **Example 1:** Clone the Column (copy)
This section explains two scenarios :\
1\. Creating a dataset from surveys.csv and incidents.csv files and create pipelines using AIOps studio functionality\
2\. Get the data from each dataset which includes some of the columns, **clone** the column X to column Y\
3\. Execute the pipeline.
### **Example 1a: Column names from surveys.csv file:**
* record\_id
* plot\_id
* length
* date etc
Step 1: \
Create and add the following pipeline code/commands into the pipeline text field.
You can copy the below code into your pipeline and execute that in your AIOps environment.\
*`##### This pipeline loads surveys.csv file into AIOps studio stores the data loaded from ##### surveys.csv file into local dataset named 'cfxdm-map-clone-column'. Once`* \
*`##### the data is saved in 'cfxdm-map-clone-column' dataset, RDA functionality`* \
*`##### '@dm:map' is used to clone columns/names from the complete dataset that`* \
*`##### was read from csv file and rename those with user defined names as follows.`*\
\
*`@files:loadfile filename = "surveys.csv"`*\
*`--> @dm:save name = 'cfxdm-map-clone-column'`*\
*`--> @dm:map from = 'record_id' & to = 'Survey_record_id'`*\
*`--> @dm:map from = 'sex' & to = 'Gender'`*
Step 2: \
Verify the above pipeline code using the 'Verify' button and execute the pipeline (as shown below)
Step 3:\
Once execution is completed, verify that the 'dm: map' cloned a column as expected (as shown below).
# Metadata - cfxdm-dm:metadata
Metadata related cfxdm functionality using cfxdm-dm:metadata
**dm:metadata:** This cfxdm bot allows the user to use this function to check/display metadata user-selected selected/loaded dataset.
This bot is useful when a user wants to analyze metadata of the selected dataset (and the type of each column within the dataset.
The current example will use "surveys.csv" provided as a dataset to explain the functionality.
**Step 1**: Download the "[surveys.csv](https://macaw-amer.s3.amazonaws.com/rda/data/surveys.csv)" file to the local machine as shown below using a standard web browser.
**Step 2**: Upload (load) the above file into your RDA system as shown below.
* Using a web browser, connect to the RDA system \:9998>
* Click on the CFXDX Python 3 icon from the main landing page of the RDA launcher (as shown below)
* Launch AIOps studio using the RDA provided text field to start AIOps studio.
**Step 3**: Upload the above downloaded CSV file into AIOps studio as shown below.
Select the upload option from the studio environment (as shown below).
Select the file and upload it into the studio environment (as shown below).
The studio will show the uploaded file in the left panel (as shown below).
**Step 4**: Create an empty dataset 'cfxdm-metadata-example" pipeline and save that in AIOps studio.
**Step 5:** Add the following pipeline code/commands into the empty pipeline text field that you have created in the above Step 4.
You can copy the below code into your pipeline and execute that in your environment.\
*`##### This pipeline loads surveys.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from surveys.csv file`*\
*`##### into local dataset named 'cfxdm-metadata'.`*\
*`##### prints the metadata from the dataset`*\
\
*`@files:loadfile filename = "surveys.csv"`*\
*`--> @dm:save name = 'cfxdm-metadata'`*\
*`--> *dm:filter`* \
*`--> @dm:metadata`*
In the above pipeline code, we added the '@dm:metadata' bot/function to the pipeline to fetch/extract metadata details of the 'cfxdm-metadata' that was created during the runtime from the surveys.csv file.
**Step 6**: Click the button 'Verify' and Click the 'Save' button (AIOps studio verifies the pipeline syntax and stores the pipeline into the earlier created 'cfxdm-metadata-example' pipeline as shown below.
**Step 7**: Users can now execute the above-created pipeline by clicking the 'Execute' button. Once the users click the 'Execute' button, AIOPs studio executes the pipeline that will load the user-defined file, save the loaded file into a local dataset named 'cfxdm-metadata' as shown below.
**Step 8**: Verify the data using AIOps Studio using 'Inspect --> Output --> Get Data'
**Step 9**: *@dm:metadata* has additional filtering capabilities using which users can query specific metadata details/data as shown below.
**Include filter** - Column name regex pattern to include in the output (include patterns are matched first and then exclude)
Following examples provide usage of include attribute using a simple regular '| (OR)' operation.
Example a:
You can copy the below code into your pipeline and execute that in your environment.\
*`##### This pipeline loads surveys.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from surveys.csv file`*\
*`##### into local dataset named 'cfxdm-metadata'.`*\
*`##### prints the metadata from the dataset using a simple regular '|'`* \
\
*`@files:loadfile filename = "surveys.csv"`*\
*`--> @dm:save name = 'cfxdm-metadata'`*\
*`--> *dm:filter`* \
*`--> @dm:metadata include = 'plot_id|record_id'`*
Once you execute the above pipeline using the 'include' filter, the output will apply the filter and show the data based on the filter (as shown below).
Verify the data using AIOps Studio using 'Inspect --> Output /Data --> Get Data'
Example b:
*`##### This pipeline loads surveys.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from surveys.csv file`*\
*`##### into local dataset named 'cfxdm-metadata'.`*\
*`##### prints the metadata from the dataset using '.*id' regular expression`*\
\
*`@files:loadfile filename = "surveys.csv"`*\
*`--> @dm:save name = 'cfxdm-metadata'`*\
*`--> *dm:filter`* \
*`--> @dm:metadata include = '.*id'`*
**Exclude filter** - Column name regex pattern to exclude in the output.
*`##### This pipeline loads surveys.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from surveys.csv file`*\
*`##### into local dataset named 'cfxdm-metadata'.`*\
*`##### prints the metadata from the dataset using '.*id' regular expression`*\
\
*`@files:loadfile filename = "surveys.csv"`*\
*`--> @dm:save name = 'cfxdm-metadata'`*\
*`--> *dm:filter`* \
*`--> @dm:metadata exclude = '.*id'`*
# Data mapping - cfxdm - dm:functions
Data manipulation and transformation using RDA functions
**dm:functions:** This cfxdm tag provides very comprehensive data manipulation & transformation functions and below are the details about them and their usage.
* **any\_non\_null:** Returns any non-null value from a list of input values, @param value is optional, if not specified, returns None when none of the listed values meet the criteria. Input must be a list (else treated as a single item list).
* **concat:** Adds prefix and suffix to the specified string, @param prefix type is string (optional). @param suffix type is a string (optional). Input must be a string. If the input is null, it is treated as ' '
* **datetime:** Parses input string and converts into an epoch milliseconds format number. Input must be a string. @param tzmap: type dict (optional). Dictionary of timezone mappings from custom/local to standard timezones. @param expr: type string (optional). A custom timestamp format with UTC/local timezone.
* **evaluate:** Given an expression evaluates the expression string. If performed on the dataframe row, it evaluates by passing the row as a dictionary. If performed on a single value, it expects an additional argument 'key' to be used in the expression. @param expr: The expression to evaluate. @param key: An optional 'key' if evaluated on a single value instead of a dictionary.
* **fixed:** Returns a fixed value specified by the 'value' parameter. @param value Type can be string or number. Input can be of any type.
* **formDecode:** Decodes input string to remove any URL encoded values. Requires no parameters. Input must be a string.
* **highest:** Returns highest non-null value from the list of integer values. @param default (optional), Type int. If provided none of the input values are non-null, returns default Input must be a number or list of numbers.
* **hours\_between:** Number of hours between two datetime strings. If only one specified, compare diff between now and that timestamp.
* **join:** Joins input list using an optional separator. @param **sep** (optional), default value is ' ' . Input is expected to be a list. If the input is not a list, it returns the value without joining.
* **jsonDecode:** Decodes input string into JSON object. Requires no parameters. Input must be a string.
* **lower**: Converts to lowercase text, requires no parameters. Input must be a string.
* **lowest**: Returns lowest non-null value from the list of int values. @param default (optional), Type int. If provided none of the input values are non-null, returns default, Input must be a number or list of numbers.
* **map\_values**: Maps input value using the specified name value dictionary. If no values match and ""\*"" key is provided, it returns the ""\*"" key's values, or else the original value will be maintained. Input must be a string.
* **match**: Matches a regular expression and extracts a specific value (if matched). @param expr Type string. Regular expression @param flags List of optional flags (A I M L S X) Input must be a string.
* **minutes\_between:** Number of minutes between two date-time strings. If only one is specified, it compares the difference between it and the current timestamp.
* **replace**: Replaces old value with new value in the input string @param oldvalue, Type string. @param new value, Type string. Input must be a string.
* **seconds\_between**: Number of seconds between two datetime strings. If only one is specified, it compares the difference between it and the current timestamp.
* **slice**: Slices a string or an array using specified indices. @param from-index Type int. Default value 0 @param to-index Type int. The default value is None. Input can be a string or a list. If neither, it converts input to a string.
* **split**: Splits the input using specified 'sep' separator. @param sep Type string. Optional. Default any whitespace characters. Input must be a string.
* **strip**: Strips white spaces from both sides of a string, Requires no parameters. Input must be a string.
* **timediff**:
* **to\_numeric**: Convert input value into numeric
* **ts\_to\_datetimestr:** Processes input number with specified '**unit**' (s,ms,ns,excel\_date) and converts the value to **datetime** string specified by '**format**', default is ISO format. Input must be a float or int. @param '**unit' (**Type string), must be s,ms,ns,excel\_date, default is '**ms**' @param '**format' (**Type string), default is None (ISO format)
* **upper**: Converts to uppercase text Requires no parameters. Input must be a string.
* **valueRef**: Extracts a specific item from the input dictionary object. @param path A dot '.' delineated path to the element within the dictionary, Input must be dictionary object.
* **when\_null**: If the specified value is null, it uses the value as per 'value' param @param value Type can be string or number. Input can be of any type.
# Any\_non\_null
### **any\_non\_null -- Function to remove any non null with default value or without default value.**
Returns any non-null value from a list of input values, @param value is optional, if not specified, returns None when none of the listed values meet the criteria. Input must be a list (else treated as a single item list).
This function uses "from" as a parameter which is usually a comma separated list of column names. using which the first non null value will be picked. If all values are Nulls. value specified via default is used (optional)
Step 1: Create an empty any\_non\_null\_example using AIOps studio as shown in the below screenshot
Step 2: Add the following pipeline code/commands into the above created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
*`##### This pipeline creates set of records (ip_address, hostname, id)`*\
*`##### using AIOps studio.`*\
\
*`##### RDA function any_non_null is used to demo this example.`*\
*`##### This function uses 'from' as a parameter which is a comma separated`*\
*`##### list of column names (from dataset), using which the first non-null`*\
*`##### value will be picked. If all the values are null in the selected`*\
*`##### columns via from, function selects value specified in default token`*\
\
*`@dm:empty`* \
*`--> @dm:addrow ipaddress = '10.10.1.1' & hostname = 'host-1-1' & id = 'a1'`*\
*`--> @dm:addrow ipaddress = '10.10.1.2' & id = 'a2'`*\
*`--> @dm:addrow ipaddress = '10.10.1.3' & id = 'a3'`*\
*`--> @dm:addrow hostname = 'host-4-4' & id = 'a4'`*\
*`--> @dm:addrow id = 'a5'`*\
*`--> @dm:map from = 'ipaddress,hostname' & to = 'ip_or_hostname' & func = 'any_non_null' & default = 'No IP_or_Hostname'`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline with out any errors (as shown below)
Step 5: RDA uses the function 'any\_non\_null' and stores the data under 'ip\_or\_hostname' column.\
As explained earlier, this functions uses 'from' as a parameter which is a comma separated list of column names (from dataset), and using that the first non-null value will be picked/added to column data/value. If all the specified columns values are null, function substitutes the value given in the default token / value (as shown below).
Note: If user does not specify any default value parameters, RDA substitutes 'None' value as default in case of any empty values from the list of column names used via 'from'
You can copy the below code into your pipeline and execute that in your environment (in this example, default token is removed from the same pipeline and ran via RDA. Once RDA executes below pipeline, output is shown in below screen capture.\
\
*`##### This pipeline creates set of records (ip_address, hostname, id)`*\
*`##### using AIOps studio.`*\
\
*`##### RDA function any_non_null is used to demo this example.`*\
*`##### This function uses 'from' as a parameter which is a comma separated`*\
*`##### list of column names (from dataset), using which the first non-null`*\
*`##### value will be picked. If all the values are null in the selected`*\
*`##### columns via from, function selects value specified in default token`*\
\
*`@dm:empty`* \
*`--> @dm:addrow ipaddress = '10.10.1.1' & hostname = 'host-1-1' & id = 'a1'`*\
*`--> @dm:addrow ipaddress = '10.10.1.2' & id = 'a2'`*\
*`--> @dm:addrow ipaddress = '10.10.1.3' & id = 'a3'`*\
*`--> @dm:addrow hostname = 'host-4-4' & id = 'a4'`*\
*`--> @dm:addrow id = 'a5'`*\
*`--> @dm:map from = 'ipaddress,hostname' & to = 'ip_or_hostname' & func = 'any_non_null'`*
# Concat
#### concat
Adds prefix and suffix to the suffix to the specified string \
@param prefix Type string. Optional. default value is ' ' )\
@param suffix Type string. Optional. default value is ' ' )\
Input must be a string. If input is null, it is treated an ' ' (empty) string.
This function allows users to concatenate column with additional string data. Additional string values can be provided as 'prefix' and or as 'suffix'. This allows users to modify column values and add additional string values as per need and requirement.
Step 1: Create an empty concat\_example using AIOps studio as shown in the below screenshot
Step 2: Add the following pipeline code/commands into the above created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a simple record for an attribute (column)`* \
*`##### using AIOps studio.`*\
\
*`##### RDA function concat is used to demo this example.`*\
*`##### This function uses 'prefix' and 'suffix' as additional string values and`* \
*`##### concatenate with original value of column X (attribute)`*\
\
*`@dm:empty`* \
*`--> @dm:addrow X = 'RDA Concat Column'`*\
*`--> @dm:map func = 'concat' & attr = 'X' & prefix = 'Hello ' & suffix = ' World'`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline with out any errors (as shown below)
Step 5: RDA uses the function to concatenate column with additional string data. In the current pipeline example, prefix is given as 'Hello' and suffix is given as 'World'. Function uses these string values and concatenates to the value of column 'X'.\
\
In the current example, attribute 'X' has a value of ''RDA Concat Column' and concat function would change that value to "**Hello RDA Concat Column World"** after successful execution of pipeline (as shown in below screenshot)
Similar to the above example, users will be able to modify the other columns (or attributes) based on the requirements.
# Datetime
**Datetime**\
Parses input string and converts into an epoch milliseconds format number. \
Input must be a string. \
@param tzmap: type dict (optional). Dictionary of timezone mappings from custom/local to standard timezones. \
@param expr: type string (optional). A custom timestamp format with UTC/local timezone.\
\
This function can be used when the datasets needs date time conversion to milliseconds or other local timezone format.
### Example 1:
Step 1: Create an empty **datetime\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a dataset row using current year.`*\
*`##### dm function datetime is used to convert year data into milliseconds`* \
\
*`@dm:empty`* \
*`--> @dm:addrow Year_To_Epoch_MS = '2021'`* \
*`--> @dm:addrow Year_To_Epoch_MS = '2031'`* \
*`--> @dm:map attr = 'Year_To_Epoch_MS' & func = 'datetime'`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm function datetime to convert year in string format to milliseconds format.\
This function will be useful when the dataset contains year(s) as column values and needed to convert to milliseconds for further time calculations.
# Date and Timestamp
### Examples:
[incidents.csv](https://macaw-amer.s3.amazonaws.com/rda/data/incidents.csv) -- This file contains a set of tickets/incident details
### ts\_to\_datetimestr - Parses input timestamp in milliseconds and converts human readable string
Step 1: Create an empty pipeine datetime\_example AIOps studio as shown in the below screenshot
Step 2: Add the following pipeline code/commands into the above created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
*`##### This pipeline creates set of records (ipaddress, timestampin milliseconds,`* \
*`##### id) using AIOps studio.`*\
\
*`##### Using RDA functionality *dm:filter renamed two column names`*\
*`##### ipaddresss to IP_Address, time to Alert_Time`*\
*`##### After renaming is completed, dm:function 'ts_to_datetimestr' is used`*\
*`#### to convert timestamp into human readable format.`*\
\
*`@dm:empty`* \
*`--> @dm:addrow ipaddress = '10.10.1.1' & time = '1623193065990' & id = 'a1'`*\
*`--> @dm:addrow ipaddress = '10.10.1.2' & time = '1623193065990' & id = 'a2'`*\
*`--> @dm:addrow ipaddress = '10.10.1.3' & time = '1623193065990' & id = 'a3'`*\
*`--> @dm:addrow ipaddress = '10.10.1.4' & time = '1623193065990' & id = 'a4'`*\
*`--> *dm:filter * get id,ipaddress as 'IP_Address',time as 'Alert_time'`*\
*`--> @dm:map attr = 'Alert_time' & func = 'ts_to_datetimestr' & unit = 'ms'`*
**Note: ts\_to\_datetimestr parameter unit (from the above) supports "ms, ns,s, excel\_date" formats.**\
\
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. Once the pipeline is executed, verify data from Alert\_time column that shows human readable time string (as shown in the below screen shot).
Step 5: Verify the data (as shown in the below screenshot)
### datetime - **Convert Human readable Date & Timestamp to Milliseconds**
Step 1: Create an empty pipeine datetime\_to\_milliseconds\_example
Step 2: Add the following pipeline code/commands into the above created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
*`##### This pipeline creates set of records (ipaddress, timestampin milliseconds,id)`* \
*`##### using AIOps studio.`*\
\
*`##### Using RDA functionality *dm:filter renamed two column names`*\
*`##### ipaddress to IP_Address, timestamp to Timestamp_in_milliseconds`*\
*`##### After renaming is completed, dm:function 'datetime' is used`*\
*`#### to convert timestamp in string format into timestamp in milliseconds`*\
\
*`@dm:empty`* \
*`--> @dm:addrow ipaddress = '10.10.1.1' & timestamp = '2021-01-15 03:00:00' & id = 'a1'`*\
*`--> @dm:addrow ipaddress = '10.10.1.2' & timestamp = '2021-01-15 04:00:00' & id = 'a2'`*\
*`--> @dm:addrow ipaddress = '10.10.1.3' & timestamp = '2021-01-15 05:00:00' & id = 'a3'`*\
*`--> @dm:addrow ipaddress = '10.10.1.4' & timestamp = '2021-01-15 06:00:00' & id = 'a4'`*\
*`--> *dm:filter * get id,ipaddress as 'IP_Address',timestamp as 'Timestamp_in_milliseconds'`*\
*`--> @dm:map attr = 'Timestamp_in_milliseconds' & func = 'datetime'`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. Once the pipeline is executed, verify data from Alert\_time column that shows human readable time string (as shown in the below screen shot).
Step 5: Verify the data (as shown in the below screenshot)
# Evaluate
**Evaluate**
Given an expression evaluates the expression string. If performed on the data frame row, it evaluates via passing the row as a dictionary. If performed on a single value, it expects an additional argument 'key' to be used in the expression. \
@param expr: The expression to evaluate. \
@param key: An optional 'key' if evaluated on a single value instead of a dictionary.
To perform arithmetic operations on columns, evaluate function is used. This function works on data type with 'Numeric' columns (Float or Integer).
{% hint style="info" %}
To perform arithmetic operations on columns, make sure the column's data type is set to '**Integer**' (Numeric) or '**Float**'. Additionally, column values should not contain **NULL** or **Empty** values.
{% endhint %}
### Example 1:
Note - This example demos the use of the 'dm: map evaluate' function using expression to calculate total VM CPU count.
Step 1: Create an empty **evaluate\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a list of rows with vm_cpu_sockets along with vm_cpu_cores`* \
*`##### for virtual machines using AIOps studio.`*\
\
*`##### RDA function 'Evaluate' is used to demo this example that uses expression`*\
*`##### to calculate total CPU allocated to each virtual machine.`*\
\
*`@dm:empty`* \
*`--> @dm:addrow vm_cpu_sockets = 1 & vm_cpu_cores = 3 & vm_id = 'vm1'`*\
*`--> @dm:addrow vm_cpu_sockets = 2 & vm_cpu_cores = 8 & vm_id = 'vm1'`*\
*`--> @dm:addrow vm_cpu_sockets = 1 & vm_cpu_cores = 2 & vm_id = 'vm1'`*\
*`--> @dm:addrow vm_cpu_sockets = 2 & vm_cpu_cores = 4 & vm_id = 'vm1'`*\
*`--> @dm:addrow vm_cpu_sockets = 2 & vm_cpu_cores = 8 & vm_id = 'vm1'`*\
*`--> @dm:addrow vm_cpu_sockets = 1 & vm_cpu_cores = 2 & vm_id = 'vm1'`*\
*`--> @dm:map to = 'vm_cpu_total' & func = 'evaluate' & expr = 'vm_cpu_sockets * vm_cpu_cores'`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the evaluate function to calculate 'vm\_cpu\_total' (using expression as shown in the pipeline code snippet) as shown in the below screenshot.
### Example 2:
Step 1: Create an empty **evaluate\_example\_2** using AIOps studio as shown in the below screenshot.
Note - In the below example, **vm\_disk\_size\_bytes** is a column that has disk capacity in bytes, using the '**evaluate**' function, create a new column '**vm\_disk\_size\_gb'** and convert the bytes into GB.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.
using the '**evaluate**' function, create a new column '**vm\_disk\_size\_gb'** and convert the bytes into GB.\
\
*`##### This pipeline creates a list of rows with vm_disk_size_bytes column that has`*\
*`##### disk capacity in bytes. Pipeline uses 'evaluate' function and maps to new column`*\
*`##### and convert the data into 'vm_disk_size_gb'`*\
\
*`##### RDA function 'Evaluate' is used to demo this example that uses expression`*\
*`##### to calculate 'vm_disk_size_gb'`*\
\
*`@dm:empty`* \
*`--> @dm:addrow vm_disk_size_bytes = 10737418240 & vm_id = 'vm1'`*\
*`--> @dm:addrow vm_disk_size_bytes = 322122542 & vm_id = 'vm2'`*\
*`--> @dm:addrow vm_disk_size_bytes = 21474836480 & vm_id = 'vm3'`*\
*`--> @dm:addrow vm_disk_size_bytes = 322122542 & vm_id = 'vm4'`*\
*`--> @dm:addrow vm_disk_size_bytes = 53687091200 & vm_id = 'vm5'`*\
*`--> @dm:map from = 'vm_disk_size_bytes' & to = 'vm_disk_size_in_gb'`*\
*`--> @dm:map attr = 'vm_disk_size_gb' & func = 'evaluate' & expr = 'vm_disk_size_gb/1024 /1024 /1024 '`* \
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the evaluate function to calculate '**vm\_disk\_size\_gb**' (using expression as shown in the pipeline code snippet) as shown in the below screenshot.
Similar to the above-provided examples, users can upload any files that require any mathematical calculation and use the 'evaluate' function/expression to print or store the data.
# Fixed
Returns a fixed value specified by the 'value' parameter.\
@param value Type can be string or number
This function is helpful when a user wants to get a fixed value from a dataset (or set of column values).
### Example 1:
Step 1: Create an empty **fixed\_example** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a list of rows with sales amount (int values) for each month`* \
*`##### using AIOps studio.`*\
\
*`##### RDA function fixed is used to demo this example (along with groupby, sum)`*\
*`##### TOTAL is used as a variable on which sum is performed to get the total sales`*\
*`##### amount for all six months (RDA uses agg = "sum" to calculate sum of all sales`*\
*`##### and provide the output of the total amount. Here TOTAL is used as an additional`*\
*`##### dummy column to allow RDA to calculate the sum.`* \
\
*`@dm:empty`* \
*`--> @dm:addrow amount = 10000 & month = 'Jan'`*\
*`--> @dm:addrow amount = 15000 & month = 'Feb'`*\
*`--> @dm:addrow amount = 11000 & month = 'Mar'`*\
*`--> @dm:addrow amount = 18000 & month = 'April'`*\
*`--> @dm:addrow amount = 8000 & month = 'May'`*\
*`--> @dm:addrow amount = 10000 & month = 'June'`*\
*`--> @dm:map to = "TOTAL" & func = "fixed" & value = "TOTAL"`*\
*`--> @dm:groupby columns = "TOTAL" & agg = "sum"`*\
*`--> *dm:filter * get TOTAL, amount`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm function 'fixed' to use a dummy column across all the available rows and provides a sum of 'amount' columns (which are defined in int') and provides an aggregate/sum value in the amount column after pipeline execution as shown in the below screenshot.
# Highest
**Highest**\
Returns highest non-null value from the list of integer values. \
@param default (optional), Type int. \
If provided none of the input values are non-null, returns default. \
Input must be a number or list of numbers.
This function is helpful when a user wants to get the highest value from a dataset (or set of column values).
### Example 1:
Step 1: Create an empty **highest\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a list of rows with various numbers (int values) for various`* \
*`##### ids using AIOps studio.`*\
\
*`##### RDA function highest is used to demo this example. In this pipeline, each row added`*\
*`##### has three columns with various numerical values. Pipeline uses RDA dm function`* \
*`##### highest to pick the highest value from set of values and prints that highest value`*\
*`##### for each row/id.`* \
\
*`@dm:empty`* \
*`--> @dm:addrow id = 'a1' & x = 10 & y = 3 & z = 9`*\
*`--> @dm:addrow id = 'a2' & x = 1 & y = 13 & z = 4`*\
*`--> @dm:addrow id = 'a3' & x = 7 & y = 3 & z = 3`*\
*`--> @dm:addrow id = 'a4' & x = 4 & y = -1 & z = 9`*\
*`--> @dm:map to = "MAX_VALUE" & from = "x,y,z" & func = 'highest'`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm function 'highest' to select the 'highest' numerical number (or value) from the dataset and map that to variable "MAX\_VALUE" and prints the output for each dataset (or row) as shown in the following screenshot.
# Join
Joins input list
**join**\
Joins input list using an optional separator. \
@param **sep** (optional), default value is ' ' . Input is expected to be a list. \
If the input is not a list, it returns the value without joining.
This function allows users to join the columns using a separator token.
### Example 1:
Step 1: Create an empty **join\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a simple record using AIOps studio.`*\
\
*`##### RDA function join is used to demo this example.`*\
*`##### This function uses different columns and join them using a separator character`*\
*`##### Function 'join' is used join the column values using a user provided separator`*\
\
*`@dm:empty`* \
*`--> @dm:addrow A = "Hello" & B = "World" & C = "Join" & D = "Example"`*\
*`--> @dm:map from = "A,B,C,D" & to = "X" & func = "join" & sep = ","`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the function 'join' to join all the column values with a separate token "," and displays the output as shown in the below screenshot.
# jsonDecode
\
Decodes input string into a JSON object. Requires no parameters. Input must be a string.
This function allows users to convert data columns with string input proper JSON object. This function helps datasets with string formats to be converted to JSON objects for further processing.
### Example 1:
Step 1: Create an empty **jsondecode\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a simple records using AIOps studio.`*\
\
*`##### RDA function jsonDecode is used to demo this example.`*\
*`##### This function uses string input of multiple rows and converts into JSON`* \
*`##### Object`*\
\
*`@dm:empty`* \
*`--> @dm:addrow attributes = '{"a":1, "b":2, "c": { "d":3, "e":4} }'`* \
*`--> @dm:addrow attributes = '{"p":11, "b":12, "c": { "d":13, "e":14} }'`* \
*`--> @dm:addrow attributes = '{"cfx":4301}'`*\
*`--> @dm:addrow attributes = '{"rda":999}'`*\
*`--> @dm:map from = 'attributes' & to = "json_attributes" & func = "jsonDecode"`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm function 'jsonDecode' to convert string attributes as shown above into JSON objects and saves under the column "json\_attributes' and prints the output for each dataset (or row) as shown in the following screenshot.
**Note: The above example uses string objects that contain numeric values.**
### Example 2:
Step 1: Create an empty **jsondecode\_example\_2** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a simple records using AIOps studio.`*\
\
*`##### RDA function jsonDecode is used to demo this example.`*\
*`##### This function uses string input of multiple rows and converts into JSON`* \
*`##### Object`*\
\
*`@dm:empty`* \
*`--> @dm:addrow attributes = '{"a":"r", "b":"d", "c":"a"}'`*\
*`--> @dm:addrow attributes = '{"cfx":"tool", "rda":"data", "use":"transformation"}'`*\
*`--> @dm:map from = 'attributes' & to = "json_attributes" & func = "jsonDecode"`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm function 'jsonDecode' to convert string attributes as shown above into JSON objects and saves under the column "json\_attributes' and prints the output for each dataset (or row) as shown in the following screenshot.
**Note: The above example uses string objects that contain string values.**
###
# Lower
**Lower**
Converts to lowercase text that require no parameters. Input must be a string.
### Example 1:
Step 1: Create an empty **lower\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a simple record using AIOps studio.`*\
\
*`##### RDA function lower is used to demo this example.`*\
*`##### This pipeline adds couple of rows with string column values`*\
*`##### with upper and mixed case strings. Uses dm function 'lower' to convert string`* \
*`##### values to all 'lower' case values.`*\
\
*`@dm:empty`* \
*`--> @dm:addrow id = 1 & col = "Hello World"`*\
*`--> @dm:addrow id = 2 & col = "CFX RDA"`*\
*`--> @dm:addrow id = 3 & col = "AIOps Solution in ML World"`*\
*`--> @dm:map attr = "col" & func = "lower"`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm function 'lower' to change the values of each column value and convert it into lower case string values and prints the output for each dataset (or row) as shown in the following screenshot.
Similar to the above example, other dataset string based columns can be converted to lower case using the dm: function 'lower'
# Lowest
**Lowest**
Returns lowest non-null value from the list of int values. \
@param default (optional), Type int. If provided none of the input values are non-null, returns default, \
Input must be a number or list of numbers.
This function is helpful when a user wants to get the lowest value from a dataset (or set of column values).
### Example 1:
Step 1: Create an empty **lowest\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a list of rows with various numbers (int values) for various`* \
*`##### ids using AIOps studio.`*\
\
*`##### RDA function lowest is used to demo this example. In this pipeline, each row added`*\
*`##### has three columns with various numerical values. Pipeline uses RDA dm function`* \
*`##### lowest to pick the lowest value from set of values and prints that value`*\
*`##### for each row/id.`* \
\
*`@dm:empty`* \
*`--> @dm:addrow id = 'a1' & x = 10 & y = 3 & z = 9`*\
*`--> @dm:addrow id = 'a2' & x = 1 & y = 13 & z = 4`*\
*`--> @dm:addrow id = 'a3' & x = 7 & y = 3 & z = 3`*\
*`--> @dm:addrow id = 'a4' & x = 4 & y = -1 & z = 9`*\
*`--> @dm:map to = "MIN_VALUE" & from = "x,y,z" & func = 'lowest'`*
\
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm function 'lowest' to select the 'lowest' numerical number (or value) from the dataset and map that to variable "MIN\_VALUE" and prints the output for each dataset (or row) as shown in the following screenshot.
Similarly, dm: function 'lowest' can be used to select the lowest value(s) from the selected dataset.
# Match
**Match**
Matches a regular expression and extracts a specific value (if matched). \
@param expression type string. Regular expression \
@param flags List of optional flags (A I M L S X) Input must be a string.\
\
I - IGNORE THE CASE\
A - ASCII\
L - LOCALE\
X - VERBOSE\
M - MULTILINE MATCH\
S - '.' matches all (otherwise regex has a limited set of chars where it matches with )
This function allows users to search for a match using regular expressions within a given dataset.
### Example 1:
Step 1: Create an empty **match\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a list of rows with hostname and ipaddress`*\
*`##### RDA dm function 'match' used to demo this example. In this pipeline, each row added`*\
*`##### has a column with IPAddress and hostname.`*\
\
*`##### Pipeline uses RDA dm function match to match to regular expression to pick only`* \
*`##### ipaddresses.`*\
\
*`@dm:empty`* \
*`--> @dm:addrow id = 'a1' & APP_Client_FQDN = "host1.acme.com"`*\
*`--> @dm:addrow id = 'a2' & APP_Client_FQDN = "host2.acme.com"`*\
*`--> @dm:addrow id = 'a3' & APP_Client_FQDN = "host3.acme.com"`*\
*`--> @dm:addrow id = 'a4' & APP_Client_FQDN = "172.17.0.1"`*\
*`--> @dm:addrow id = 'a5' & APP_Client_FQDN = "192.168.60.120"`*\
*`--> @dm:addrow id = 'a6' & APP_Client_FQDN = "172.17.0.4"`*\
*`--> @dm:addrow id = 'a7' & APP_Client_FQDN = "172.17.0.3"`*\
*`--> @dm:map func = 'match' & from = 'APP_Client_FQDN' & to = 'IPaddress' & expr = "^(?:[0-9]{1,3}.){3}[0-9]{1,3}$"`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the 'match' function to match a regular expression from a selected column, picks up that value, applies regular expression, stores it in the new column (IP Address), and prints it to the screen as shown below.
### Example 2:
Step 1: Create an empty **match\_example\_2** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a list of rows with hostname and ipaddress`*\
*`##### RDA dm function 'match' used to demo this example. In this pipeline, each row added`*\
*`##### has a column with IPAddress and hostname.`*\
\
*`##### Pipeline uses RDA dm function match to match to regular expression to pick only`* \
*`##### hostname from FQDN from the dataset.`*\
\
*`@dm:empty`* \
*`--> @dm:addrow id = 'a1' & APP_Client_FQDN = "host1.acme.com"`*\
*`--> @dm:addrow id = 'a2' & APP_Client_FQDN = "host2.acme.com"`*\
*`--> @dm:addrow id = 'a3' & APP_Client_FQDN = "host3.acme.com"`*\
*`--> @dm:addrow id = 'a4' & APP_Client_FQDN = "172.17.0.1"`*\
*`--> @dm:addrow id = 'a5' & APP_Client_FQDN = "192.168.60.120"`*\
*`--> @dm:addrow id = 'a6' & APP_Client_FQDN = "172.17.0.4"`*\
*`--> @dm:addrow id = 'a7' & APP_Client_FQDN = "172.17.0.3"`*\
*`--> @dm:map func = 'match' & from = 'APP_Client_FQDN' & to = 'Hostname' & expr = "(.*).acme.com"`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the 'match' function to match a regular expression from a selected column, picks up that value, applies regular expression, stores it in the new column (hostname), and prints it to the screen as shown below.
### Example 3:
Step 1: Create an empty **match\_example\_3** using AIOps studio as shown in the below screenshot.
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a list of rows with hostname and ipaddress`*\
*`##### RDA dm function 'match' used to demo this example. In this pipeline, each row added`*\
*`##### has a column with IPAddress and hostname.`*\
\
*`##### Pipeline uses RDA dm function match to match to regular expression to pick FQDN`* \
*`##### and excluding IP Addresses from the dataset.`*\
\
*`@dm:empty`* \
*`--> @dm:addrow id = 'a1' & APP_Client_FQDN = "host1.acme.com"`*\
*`--> @dm:addrow id = 'a2' & APP_Client_FQDN = "host2.acme.com"`*\
*`--> @dm:addrow id = 'a3' & APP_Client_FQDN = "host3.acme.com"`*\
*`--> @dm:addrow id = 'a4' & APP_Client_FQDN = "172.17.0.1"`*\
*`--> @dm:addrow id = 'a5' & APP_Client_FQDN = "192.168.60.120"`*\
*`--> @dm:addrow id = 'a6' & APP_Client_FQDN = "172.17.0.4"`*\
*`--> @dm:addrow id = 'a7' & APP_Client_FQDN = "172.17.0.3"`*\
*`--> @dm:map func = 'match' & from = 'APP_Client_FQDN' & to = 'Hostname' & expr =`* \
*`"^(?!:\/\/)(?=.{1,255}$)((.{1,63}\.){1,127}(?![0-9]*$)[a-z0-9-]+\.?)$"`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the 'match' function to match a regular expression from a selected column, picks up that value, applies regular expression, stores it in the new column (FQDN - hostname.domain), and prints it to the screen as shown below.
Note: Each of the above examples is using different expressions (regular expressions) to extract patterns to match criteria and output required data from the dataset. Also, you can remove the rows with 'None' values using the dm function "@dm:fixnull columns = 'Hostname'"
# Minutes\_Between
**Minutes\_between**
The number of minutes between two date-time strings. If only one is specified, it compares the difference between it and the current timestamp.
### Example 1:
Step 1: Create an empty **minutes\_between\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a simple time fields time1 and time2 with two distinctive`*\
*`##### values as a dataset.`* \
*`##### Pipeline uses RDA dm function minutes_between to calculate minutes between two`*\
*`##### timestamps and prints the value. In addition, RDA dm function ts_to_datetimestr`*\
*`##### is used to convert the the time difference into string (other units are ms,ns,`* \
*`##### excel_date)`*\
\
\
*`@dm:empty`* \
*`--> @dm:addrow time1 = 1623193065990 & time2 = 1623193085990`*\
*`--> @dm:map to = 'Change_In_Minutes' & from = 'time1,time2' & func = 'minutes_between'`*\
*`--> @dm:map attr = 'Change_In_Minutes' & func = 'ts_to_datetimestr' & unit = 's'`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm function minutes\_between to calculate the difference between two timestamps and presents the output as shown below screenshot. \
Note: In addition to the above dm function, the pipeline also uses ts\_to\_datetimestr to convert the timestamp value to a string.
# Replace
### Replace
Replaces old value with new value in the input string @param old value, Type string. @param new value, Type string. Input must be a string.
Replace 'oldvalue' with 'newvalue' in the input string\
@param oldvalue Type string \
@param newvalue Type string\
Input must be a string
This function allows users to replace or substitute a new string value in place of an old existing string value. This allows users to modify column values and add additional string values as per need and requirement
### Example 1:
Step 1: Create an empty **replace\_example** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a simple record for an attribute (column)`* \
*`##### using AIOps studio.`*\
\
*`##### RDA function replace is used to demo this example.`*\
*`##### This function uses 'newvalue' to replace 'oldvalue' both are in`* \
*`##### string format.`*\
*`##### Function 'replace' replaces old value with new value`*\
\
*`@dm:empty`* \
*`--> @dm:addrow id = '1' & category = 'Cluster-1' & value = 10`*\
*`--> @dm:addrow id = '2' & category = 'Cluster-2' & value = 101`*\
*`--> @dm:addrow id = '3' & category = 'Cluster-3' & value = 21`*\
*`--> @dm:addrow id = '4' & category = 'Cluster-4' & value = 30`*\
*`--> @dm:addrow id = '5' & category = 'Test-Cluster-5' & value = 500`*\
*`--> @dm:addrow id = '6' & category = 'Cluster-6' & value = 50`*\
*`--> @dm:map func = 'replace' & oldvalue = 'Cluster' & newvalue = 'CFX-Cluster' & attr = 'category'`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the replace function to replace the old value with a new value for the selected column. Click on Verify to check replaced values as shown below.
Simple Example - The following is a simple example of another example and final verify output where attribute 'column1' has a value of "Hello abc World" is replaced with "Hello CFX World" after successful execution of pipeline. Try this on your own (as shown below screenshot).
### Example 2:
This example provides a pipeline that uses the 'replace' function to replace a specific character within a string. This will be useful in case if a user wants to search and replace a special character or add additional tokens etc.
Step 1: Create an empty **replace\_char\_example** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a simple record for an attribute (column)`* \
*`##### using AIOps studio.`*\
\
*`##### RDA function replace is used to demo this example.`*\
*`##### This function uses 'newvalue' to replace 'oldvalue' both are in`* \
*`##### string format.`*\
*`##### Function 'replace' replaces char '"' with empty char ''`*\
\
*`@dm:empty`* \
*`--> @dm:addrow column = 'Hello " World'`*\
*`--> @dm:eval column = "column.replace(chr(34),'')"`*
*Note: Users can use this example to try other characters replacements and/or other character token replacements*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the replace function to replace a character from the string with an empty token and prints the output of the original string as shown in the below screenshot.
Old value --> **'Hello " World'**\
New value --> **'Hello World'**
### Example 3:
This example provides a pipeline that uses the 'replace' function multiple times to search for a pattern that is repeated within a string and replace a common character separator or token within that string.
Step 1: Create an empty **replace\_multi\_char\_example** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a simple record for an attribute (mac)`* \
*`##### using AIOps studio.`*\
\
*`##### RDA function replace is used twice to demo this example.`*\
*`##### This function uses 'newvalue' to replace 'oldvalue' both are in`* \
*`##### char format.`*\
*`##### Function 'replace' replaces char '| ' and '|' with ',' character`* \
\
*`@dm:empty`* \
*`--> @dm:addrow mac = '001b 6384 45e6| 001c 4432 4467 000a 959d 6816|000a 959d 6816'`* \
*`--> @dm:eval mac = "mac.replace('| ',',').replace('|',',')"`*
*Note: Example uses two patterns to replace special character or token with other tokens within the mac addresses*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the function 'replace' to replace multi-pattern character tokens, adds 'comma' as token, and returns the string after replacement as shown in the below screenshot.
In addition to the above-provided examples, users can use replace for any substitution of strings, special chars, within the pipeline context and store the data.
### Example 4:
This example provides a pipeline that uses the 'replace' function when the column values have ’Cluster” as the name should add a prefix (e.g. Sub or other prefixes ). This will help users to replace or substitute a string value with custom strings or prefixes.
Step 1: Create an empty **replace\_prefix\_example** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates set of records that contains Cluster-x as value strings`*\
*`##### in the dataset.`*\
\
*`##### RDA function replace is used to demo this example.`*\
*`##### This function uses replace (substitute) column values that contain Cluster`* \
*`##### with a prefix 'Sub' and adds it back`*\
*`##### Function 'replace' replaces old value (Cluster) with new value (Sub)`*\
\
*`@dm:empty`* \
*`--> @dm:addrow id = '1' & category = 'Cluster-1' & value = 10`*\
*`--> @dm:addrow id = '2' & category = 'Cluster-2' & value = 101`*\
*`--> @dm:addrow id = '3' & category = 'Cluster-3' & value = 21`*\
*`--> @dm:addrow id = '4' & category = 'Cluster-4' & value = 30`*\
*`--> @dm:addrow id = '5' & category = 'Test-Cluster-5' & value = 500`*\
*`--> @dm:addrow id = '6' & category = 'Cluster-6' & value = 50`*\
*`--> @dm:map func = 'replace' & oldvalue = 'Cluster' & newvalue = 'CFX-Cluster' & attr = 'category'`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the function 'replace' to replace/substitute the 'Cluster' string with 'CFX' and returns the string after replacement as shown in the below screenshot.
# Seconds\_Between
The number of seconds between two date-time strings. If only one is specified, it compares the difference between it and the current timestamp.
### Example 1:
Step 1: Create an empty **seconds\_between\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a simple time fields time1 and time2 with two distinctive`*\
*`##### values as a dataset.`* \
*`##### Pipeline uses RDA dm function seconds_between to calculate seconds between two`*\
*`##### timestamps and prints the value. In addition, RDA dm function ts_to_datetimestr`*\
*`##### is used to convert the the time difference into string (other units are ms,ns,`* \
*`##### excel_date)`*\
\
\
*`@dm:empty`* \
*`--> @dm:addrow time1 = 1623193065990 & time2 = 1623193085990`*\
*`--> @dm:map to = 'Change_In_Seconds' & from = 'time1,time2' & func = 'seconds_between'`*\
*`--> @dm:map attr = 'Change_In_Seconds' & func = 'ts_to_datetimestr' & unit = 's'`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm function seconds\_between to calculate the difference between two timestamps and presents the output as shown below screenshot. \
Note: In addition to the above dm function, the pipeline also uses ts\_to\_datetimestr to convert the timestamp value to a string.
# Slice
**Slice**\
Slices a string or an array using specified indices. \
@param from-index Type int (Default value 0) \
@param to-index Type int. \
Input can be a string or a list. If neither, it converts input to a string.
This function allows users to selectively pick a sub-string from a string.
### Example 1:
Step 1: Create an empty **slice\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates an ipaddress list with different IP addresses as string`*\
*`##### in a dataset.`*\
*`##### Pipeline uses RDA dm slice select first and second IP addresses and prints`*\
*`##### the values.`*\
*`##### In addition, RDA dm filter function is used to print initial list along with`*\
*`##### sliced values.`*\
\
*`@dm:empty`* \
*`--> @dm:addrow IPV4_Address_List = '10.95.122.15,10.95.122.16,10.95.122.107'`*\
*`--> @dm:map from = "IPV4_Address_List" & to = "First_IP" & func = 'slice' & toIdx = 12`* \
*`--> @dm:map from = "IPV4_Address_List" & to = "Second_IP" & func = 'slice' & fromIdx = 13 & toIdx = 25`*\
*`--> *dm:filter * get IPV4_Address_List, First_IP, Second_IP`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm function slice to pick the selected IP\_Address as per the index that was provided in the pipeline. In the first slice logic, default from the index (default 0) is selected whereas, in the second line, explicit indices are provided to select/slice the required IP Address. Once the IP list is sliced for the first IP Address and second IP Address, it is printed to the output (as shown below)\
\
Note: In addition to the above dm function, the pipeline also uses a dm filter function to select the output.
# Split
S**plit**\
Splits the input using the specified 'sep' separator. \
@param sep Type string. Optional ( Default any whitespace characters) \
Input must be a string.\
\
This function allows users to split the string using a token that is provided by the user.
### Example 1:
Step 1: Create an empty **split\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates an ipaddress list with different IP addresses as string`*\
*`##### in a dataset.`*\
*`##### Pipeline uses RDA dm split the string into three elements using split token ','`*\
*`##### Once the string is split, output is displayed.`*\
\
*`@dm:empty`* \
*`--> @dm:addrow IPV4_Address_List = '10.95.122.15,10.95.122.16,10.95.122.107'`*\
*`--> @dm:map from = 'IPV4_Address_List' & to = 'IP_V4_Address' & func = 'split' & sep = ','`* \
*`--> *dm:filter * get IPV4_Address_List, IP_V4_Address`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm function split to split the string into multiple elements or tokens based on the separator that was provided by the user.\
\
Once the IP list is split using a 'sep' token, the output is stored in another IP\_V4\_Address element.\
\
Note: In addition to the above dm function, the pipeline also uses a dm filter function to select and display the output.
# Strip
**Strip**\
Strips white spaces from both sides of a string, \
Requires no parameters. \
Input must be a string.
This function allows users to strip additional or special characters from the input string.
### Example 1:
Step 1: Create an empty **strip\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates two rows with different strings that contain additional white`*\
*`##### spaces at the end of the strings and is stored as dataset.`*\
*`##### Pipeline uses RDA dm strip function to remove additional white spaces at the end`*\
*`##### and output is displayed without white spaces.`*\
\
*`@dm:empty`* \
*`--> @dm:addrow Input_String = 'RDA is a data exchange and transformation tool '`*\
*`--> @dm:addrow Input_String = 'AIOps uses RDA tool '`* \
*`--> @dm:map from = 'Input_String' & to = 'Output_String' & func = strip`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm function strip to remove white spaces from the input string. In this example, two strings with additional white spaces are fed to the dm function 'strip' which in turn removes white spaces and prints the output. This function will be useful when the dataset contains strings with white spaces (tabs, etc).
# To\_Numeric
**To\_Numeric**\
\
Convert input value into numeric
### Example 1:
Step 1: Create an empty **to\_numeric\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates disk size in bytes as a dataset row. Pipeline uses`*\
*`##### dm function to_numeric to convert string data into numeric. Next uses`*\
*`##### mathematical operation to convert bytes to GB and displays both bytes`* \
*`##### and Gb to the output`*\
\
*`@dm:empty`* \
*`--> @dm:addrow disk_size_in_bytes = '10737418240'`*\
*`--> @dm:map attr = 'disk_size_in_bytes' & func = 'to_numeric'`* \
*`--> @dm:map from = 'disk_size_in_bytes' & to = 'disk_size_in_GB' & func = evaluate & expr = "disk_size_in_bytes/1024/1024/1024"`*\
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm function to\_numeric to convert string data into numeric, which in turn is used to convert disk size that was in bytes to convert to GB as shown in the output.\
This function will be useful when the dataset contains strings that need to be converted to numeric and perform some mathematical logic similar to the above-explained example.
# Ts\_To\_Datetimestr
**Ts\_To\_Datetimestr**\
Processes input number with specified '**unit**' (s, ms, ns, excel\_date) and converts the value to **datetime** string specified by '**format**', default is ISO format. \
Input must be a float or int. \
@param '**unit' (**Type string), must be s,ms,ns,excel\_date, default is '**ms**' \
@param '**format' (**Type string), default is None (ISO format)\
\
This function is useful in case users want to convert to proper date-time strings.
### Example 1:
Step 1: Create an empty **ts\_to\_datetimestr\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a dataset row using current time in milliseconds.`*\
*`##### dm function ts_to_datetimestr is used to convert string data into proper`* \
*`##### date time string using ISO format (default). This function takes an additional`* \
*`##### parameter unit using 'ms', 's'.`* \
\
*`@dm:empty`* \
*`--> @dm:addrow Time_in_Milliseconds = '1626147743837'`*\
*`--> @dm:map to = 'Time_in_String' & from = 'Time_in_Milliseconds' & func = 'ts_to_datetimestr' & unit = 'ms'`* \
*`--> *dm:filter * get Time_in_String`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm function **ts\_to\_datetimestr** to convert milliseconds into readable string format using ISO format as shown below.\
\
This function will be useful when the dataset contains timestamps in ms, s, excel\_date that need to be converted to string format for reporting purposes.
# Upper
U**pper**\
Converts to uppercase text and require no parameters. Input must be a string.
This function is useful when string inputs of a dataset need to be converted to uppercase.
### Example 1:
Step 1: Create an empty **upper\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a simple record using AIOps studio.`*\
\
*`##### RDA function upper is used to demo this example.`*\
*`##### This pipeline adds couple of rows with string column values`*\
*`##### with upper and mixed case strings. Uses dm function 'upper' to convert string`* \
*`##### values to all 'upper' case values.`*\
\
*`@dm:empty`* \
*`--> @dm:addrow id = 1 & col = "Hello World"`*\
*`--> @dm:addrow id = 2 & col = "CFX RDA"`*\
*`--> @dm:addrow id = 3 & col = "AIOps Solution in ML World"`*\
*`--> @dm:map attr = "col" & func = "upper"`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm function 'upper' to change the values of each column value and convert it into upper case string values and prints the output for each dataset (or row) as shown in the following screenshot.
# When\_Null
**when\_null**\
If the specified value is null, it uses the value as per 'value' \
param @param value Type can be string or number. Input can be of any type.
This function can be used when the dataset needs a default value when the dataset column value has null values and the user wants to substitute a default value.
#### Example 1:
Step 1: Create an empty **when\_null\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a set of records with IP Addresses and hostnames (with`*\
*`##### some null values for demonstration`*\
*`##### RDA function when_null is used to demo this example.`*\
\
*`##### This pipeline adds couple of rows with IP Addresses and hostnames`* \
*`##### with some null values. Uses dm function 'when_null' to check for any`*\
*`##### null values for hostnames and substitute localhost as default value`*\
*`##### Also, 127.0.0.1 as default when an IP address is null`*\
\
*`@dm:empty`* \
*`--> @dm:addrow ipaddress = '10.10.1.1' & hostname = 'host-1-1' & id = 'a1'`*\
*`--> @dm:addrow ipaddress = '10.10.1.2' & id = 'a2'`*\
*`--> @dm:addrow ipaddress = '10.10.1.3' & id = 'a3'`*\
*`--> @dm:addrow hostname = 'host-4-4' & id = 'a4'`*\
*`--> @dm:addrow id = 'a5'`*\
*`--> @dm:map from = 'hostname' & to = 'Hostname' & func = 'when_null' & value = 'localhost'`*\
*`--> @dm:map from = 'ipaddress' & to = 'IP Address' & func = 'when_null' & value = '127.0.0.1'`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm function 'when\_null' to substitute the default value specified by the user for both hostname (as localhost) and IP Address (as 127.0.0.1) and prints the output for each dataset (or row) as shown in the following screenshot.
This function will be useful when the dataset column values are nulls and the user wants to substitute with a default value for further processing in the pipeline.
# Data Mapping cfxdm - dm:sort
dm:sort is the default sort (text based)
**dm:sort:** This cfxdm tag allows the user to sort the data for a given column(s) in ascending or descending order.
**dm: sort syntax:** It supports the below arguments
* **columns (mandatory)**. It accepts one or more column names. If more than one column is specified, use a comma as a separator.
* **order (optional)**. Supported values are '**ascending**' or '**descending**'. When not specified, the default applied sort order is 'ascending'
**dm:sort columns = 'COLUMN\_A,COLUMN\_B,..'**
OR
**dm:sort columns = 'COLUMN\_A,COLUMN\_B,..' & order = 'ascending/descending'**
### Example 1:
Sort the selected column. (**ascending** order)
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a set of records/dateset details coming from a vCenter`* \
*`##### environment. This dataset includes datastore details, Folder name, guest_hostname,`* \
*`##### guest_ip_address.`*\
\
*`##### This pipeline uses dm sort function to sort the selected columns in ascending`* \
*`##### order and prints the output.`*\
\
*`@dm:empty`* \
*`--> @dm:addrow datastore = 'CFX-QA-Store-NFS-qnap' & Folder = '' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'test2.oia.cloudfabrix.com' & guest_ip_address = '10.95.134.17'`* \
\
*`--> @dm:addrow datastore = 'CFX-QA-Store-NFS-qnap' & Folder = 'User VMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'qa.oia.cloudfabrix.com' & guest_ip_address = '10.95.122.13'`*\
\
*`--> @dm:addrow datastore = 'ENG_ISOs,netapp-qa-nfs' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'Ubuntu Linux (64 bit)' & guest_hostname = 'ubuntuqa' & guest_ip_address = '10.95.102.172'`*\
\
*`--> @dm:addrow datastore = 'datastore1 (6)' & Folder = 'Ravi-Pisupati' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'elkstack' & guest_ip_address = '10.95.121.218'`*\
\
*`--> @dm:addrow datastore = 'datastore1-198' & Folder = '' & guest_full_name = 'Microsoft Windows Server' & guest_hostname = 'spdnode.adwinstack' & guest_ip_address = '10.95.132.4'`* \
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-01' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'localhost' & guest_ip_address = '10.95.103.115'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-01' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'localhost' & guest_ip_address = '10.95.103.115'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'democloudp' & guest_ip_address = '10.95.122.212'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'ravip-v201-platform' & guest_ip_address = '10.95.122.216'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'cfxautomatesvc' & guest_ip_address = '10.95.125.116'`*\
\
*`--> @dm:sort columns = 'datastore'`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm sort function to perform the default sort of the column values (text-based) \
and prints the output for each dataset (or row) as shown in the following screenshot.
### Example 2:
Sort the selected column. (**descending** order)
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a set of records/dateset details coming from a vCenter`* \
*`##### environment. This dataset includes datastore details, Folder name, guest_hostname,`* \
*`##### guest_ip_address.`*\
\
*`##### This pipeline uses dm sort function to sort the selected columns in ascending`* \
*`##### order and prints the output.`*\
\
*`@dm:empty`* \
*`--> @dm:addrow datastore = 'CFX-QA-Store-NFS-qnap' & Folder = '' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'test2.oia.cloudfabrix.com' & guest_ip_address = '10.95.134.17'`* \
\
*`--> @dm:addrow datastore = 'CFX-QA-Store-NFS-qnap' & Folder = 'User VMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'qa.oia.cloudfabrix.com' & guest_ip_address = '10.95.122.13'`*\
\
*`--> @dm:addrow datastore = 'ENG_ISOs,netapp-qa-nfs' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'Ubuntu Linux (64 bit)' & guest_hostname = 'ubuntuqa' & guest_ip_address = '10.95.102.172'`*\
\
*`--> @dm:addrow datastore = 'datastore1 (6)' & Folder = 'Ravi-Pisupati' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'elkstack' & guest_ip_address = '10.95.121.218'`*\
\
*`--> @dm:addrow datastore = 'datastore1-198' & Folder = '' & guest_full_name = 'Microsoft Windows Server' & guest_hostname = 'spdnode.adwinstack' & guest_ip_address = '10.95.132.4'`* \
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-01' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'localhost' & guest_ip_address = '10.95.103.115'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-01' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'localhost' & guest_ip_address = '10.95.103.115'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'democloudp' & guest_ip_address = '10.95.122.212'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'ravip-v201-platform' & guest_ip_address = '10.95.122.216'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'cfxautomatesvc' & guest_ip_address = '10.95.125.116'`*\
\
*`--> @dm:sort columns = 'datastore' & order = 'descending'`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm sort function to perform the default sort of the column values (text-based) \
in descending order and prints the output for each dataset (or row) as shown in the following screenshot.
### Example 3:
Sort the selected multiple columns. (**descending** order)
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a set of records/dateset details coming from a vCenter`* \
*`##### environment. This dataset includes datastore details, Folder name, guest_hostname,`* \
*`##### guest_ip_address.`*\
\
*`##### This pipeline uses dm sort function to sort the selected columns in ascending`* \
*`##### order and prints the output.`*\
\
*`@dm:empty`* \
*`--> @dm:addrow datastore = 'CFX-QA-Store-NFS-qnap' & Folder = '' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'test2.oia.cloudfabrix.com' & guest_ip_address = '10.95.134.17'`* \
\
*`--> @dm:addrow datastore = 'CFX-QA-Store-NFS-qnap' & Folder = 'User VMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'qa.oia.cloudfabrix.com' & guest_ip_address = '10.95.122.13'`*\
\
*`--> @dm:addrow datastore = 'ENG_ISOs,netapp-qa-nfs' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'Ubuntu Linux (64 bit)' & guest_hostname = 'ubuntuqa' & guest_ip_address = '10.95.102.172'`*\
\
*`--> @dm:addrow datastore = 'datastore1 (6)' & Folder = 'Ravi-Pisupati' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'elkstack' & guest_ip_address = '10.95.121.218'`*\
\
*`--> @dm:addrow datastore = 'datastore1-198' & Folder = '' & guest_full_name = 'Microsoft Windows Server' & guest_hostname = 'spdnode.adwinstack' & guest_ip_address = '10.95.132.4'`* \
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-01' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'localhost' & guest_ip_address = '10.95.103.115'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-01' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'localhost' & guest_ip_address = '10.95.103.115'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'democloudp' & guest_ip_address = '10.95.122.212'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'ravip-v201-platform' & guest_ip_address = '10.95.122.216'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'cfxautomatesvc' & guest_ip_address = '10.95.125.116'`*\
\
*`--> @dm:sort columns = 'datastore,guest_full_name' & order = 'descending'`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm sort function to perform the default sort of the multiple column values (text-based) \
in descending order and prints the output for each dataset (or row) as shown in the following screenshot.
# Data Mapping cfxdm - dm:head
**dm:head:** This cfxdm tag allows the user to fetch top 'n' rows from the queried data.
**dm: head** syntax**:**
* **n (optional)**. Specify the number of top rows that need to be listed. When this argument is not specified, by default it retrieves the top 10 rows.
\
This section explains how users can use a CSV file loaded into a dataset. This saved dataset will be used to explain how the dm: head function can be used to check the head of the stored dataset.
{% hint style="info" %}
Download the [incidents.csv](https://macaw-amer.s3.amazonaws.com/rda/data/incidents.csv) file to the local machine as shown below using a standard web browser.
{% endhint %}
### Example 1:
Default dm: head functionality is captured in this example.
Step 1: Download 'incidents.csv' to the AIOps RDA environment as shown below from the local file system.
Step 2**:** Upload the file 'incidents.csv' to AIOps studio using file-browser (as shown below)
Step 3**:** Add a new empty pipeline with the name "**dm\_head\_example\_1**" as shown below and click the "Save" button (this step will create an empty pipeline and saves it to AIOps studio).
Step 4: Add the following pipeline commands into the empty pipeline text field that you have created in above Step 3.
You can copy the below code into your pipeline and execute that in your environment.\
*`##### This pipeline loads incidents.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from incidents.csv file`*\
*`##### into local dataset named 'incident-summary'.`*\
*`##### prints the data that was stored`*\
*`@files:loadfile filename = "incidents.csv"`*\
*`--> @dm:save name = 'incidents-summary'`*\
*`--> *dm:filter *`*
Step 5: Check the data from incidents.csv by executing the pipeline and verifying using inspect data as shown below (screenshot -1 & screenshot-2)
Note: There are 436 Rows stored in 'incidents-summary' dataset that was loaded from the incidents.csv file.
\
Step 6: Now, add the following additional pipeline code to use the dm: head function to the previously created pipeline from Step-4 as shown below (Edit and add the following pipeline code) and click verify to verify the pipeline code as shown below.
*`##### This pipeline loads incidents.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from incidents.csv file`*\
*`##### into local dataset named 'incident-summary'.`*\
*`##### prints the data that was stored`*\
*`@files:loadfile filename = "incidents.csv"`*\
*`--> @dm:save name = 'incidents-summary'`*\
*`--> *dm:filter *`* \
*`--> @dm:head`*
Step 7**:** Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 8: RDA uses the dm head function to perform the selection of top '10' rows and prints to output as shown below. In addition, it displays the number of rows that were selected by default "dm: head" function that was run on the dataset stored off-of incidents.csv file.
### Example 2:
dm: head functionality with and additional argument '**n (optional)'** is captured in this example.
Repeat the step 'Step-1, Step-2, Step-3 as explained in Example -1.
In Step-3, add a new empty pipeline with the name "**dm\_head\_example\_2**" as shown below and click the "Save" button (this step will create an empty pipeline and saves it to AIOps studio).
Step 4: Now, add the following additional pipeline code to use the dm: head function to the previously created empty pipeline from Step-3 as shown below (Edit and add the following pipeline code) and click verify to verify the pipeline code as shown below.
*`##### This pipeline loads incidents.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from incidents.csv file`*\
*`##### into local dataset named 'incident-summary'.`*\
*`##### prints the data that was stored`*\
*`@files:loadfile filename = "incidents.csv"`*\
*`--> @dm:save name = 'incidents-summary'`*\
*`--> *dm:filter *`* \
*`--> @dm:head n = 20`*
Step 5**:** Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 6: RDA uses the dm head function to perform the selection of top '20' rows and prints to output as shown below. In addition, it displays the number of rows that were selected by default "dm: head" function that was run on the dataset stored off-of incidents.csv file.
# Data Mapping cfxdm - dm: tail
**dm:tail:** This cfxdm tag allows the user to fetch the last 'n' rows from the queried data.
**dm: tail** syntax**:**
* **n (optional)**. Specify the number of last rows that need to be listed. When this argument is not specified, by default it retrieves the last 10 rows.
This section explains how users can use a CSV file loaded into a dataset. This saved dataset will be used to explain how the dm: tail function can be used to check the tail of the stored dataset.
{% hint style="info" %}
Download the [incidents.csv](https://macaw-amer.s3.amazonaws.com/rda/data/incidents.csv) file to the local machine as shown below using a standard web browser
{% endhint %}
{% hint style="info" %}
For predictable results, use it along with the **dm: sort** tag (Example -2 )
{% endhint %}
### Example 1:
Default dm: tail functionality is captured in this example.
Step 1: Download 'incidents.csv' to the AIOps RDA environment as shown below from the local file system.
Step 2**:** Upload the file 'incidents.csv' to AIOps studio using file-browser (as shown below)
Step 3**:** Add a new empty pipeline with the name "**dm\_tail\_example\_1**" as shown below and click the "Save" button (this step will create an empty pipeline and saves it to AIOps studio).
Step 4: Add the following pipeline commands into the empty pipeline text field that you have created in above Step 3.
You can copy the below code into your pipeline and execute that in your environment.\
*`##### This pipeline loads incidents.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from incidents.csv file`*\
*`##### into local dataset named 'incident-summary'.`*\
*`##### prints the data that was stored`*\
*`@files:loadfile filename = "incidents.csv"`*\
*`--> @dm:save name = 'incidents-summary'`*\
*`--> *dm:filter *`*
Step 5: Check the data from incidents.csv by executing the pipeline and verifying using inspect data as shown below (screenshot -1 & screenshot-2)
Step 6: Now, add the following additional pipeline code to use the dm: tail function to the previously created pipeline from Step-4 as shown below (Edit and add the following pipeline code) and click verify to verify the pipeline code as shown below.
*`##### This pipeline loads incidents.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from incidents.csv file`*\
*`##### into local dataset named 'incident-summary'.`*\
*`##### prints the data that was stored`*\
*`@files:loadfile filename = "incidents.csv"`*\
*`--> @dm:save name = 'incidents-summary'`*\
*`--> *dm:filter *`* \
*`--> @dm:tail`*
Step 7**:** Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 8: RDA uses the dm tail function to perform the selection of last/tail '10' rows and prints to output as shown below. In addition, it displays the number of rows that were selected by default "dm: tail" function that was run on the dataset stored off-of incidents.csv file.
### Example 2:
Default dm: tail functionality is captured in this example using sort functionality
Step 3**:** Add a new empty pipeline with the name "**dm\_tail\_example\_2**" as shown below and click the "Save" button (this step will create an empty pipeline and saves it to AIOps studio) \
and add the following pipeline commands into the pipeline text field.
You can copy the below code into your pipeline and execute that in your environment.\
*`##### This pipeline loads incidents.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from incidents.csv file`*\
*`##### into local dataset named 'incident-summary'.`*\
*`##### prints the data that was stored`*\
*`@files:loadfile filename = "incidents.csv"`*\
*`--> @dm:save name = 'incidents-summary'`*\
*`--> *dm:filter *`* \
*`--> dm:sort columns = 'Summary'`*\
*`--> dm:tail`*
Step 4**:** Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm tail function to perform the selection of bottom rows and prints to output as shown below (also using dm: sort). In addition, it displays the number of rows that were selected by default "dm: tail" function that was run on the dataset stored off-of incidents.csv file.
# Data Mapping cfxdm - dm: dedup
**dm: dedup:** This cfxdm tag allows the user to remove the duplicate values from the queried data for a selected column or columns.
It can be used to find the unique values from a selected column or unique values from more than one selected column by looking at them as a combined value.
**dm: dedup** syntax**:**
* **columns (**optional**)**. Specify a column or columns (comma separated) on which de-duplication of the data is to be applied.
This section explains how users can use a CSV file loaded into a dataset. This saved dataset will be used to explain how the dm: dedup function can be used to check the dedup of the stored dataset.
{% hint style="info" %}
Download the [incidents.csv](https://macaw-amer.s3.amazonaws.com/rda/data/incidents.csv) file to the local machine as shown below using a standard web browser
{% endhint %}
### Example 1:
Default dm: dedupe functionality is captured in this example.
Example-1 captures dedupe functionality for an inline dataset.
Step 1: Create an empty **dm\_dedupe\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a set of records with duplicate IP Addresses and hostnames`*\
*`##### RDA function dm:dedup is used to demo this example.`*\
\
*`##### This pipeline adds couple of rows with duplicate IP Addresses and hostnames`* \
*`##### Uses dm function 'dedup' to remove duplicate values from IP Addresses and hostnames`*\
\
*`@dm:empty`* \
*`--> @dm:addrow ipaddress = '10.10.1.1' & hostname = 'host-1-1' & id = 'a1'`*\
*`--> @dm:addrow ipaddress = '10.10.1.2' & id = 'a2'`*\
*`--> @dm:addrow ipaddress = '10.10.1.2' & id = 'a3'`*\
*`--> @dm:addrow ipaddress = '10.10.1.3' & id = 'a4'`*\
*`--> @dm:addrow ipaddress = '10.10.1.3' & id = 'a5'`*\
*`--> @dm:addrow hostname = 'host-4-4' & id = 'a6'`*\
*`--> @dm:addrow hostname = 'host-4-4' & id = 'a7'`*\
*`--> @dm:addrow id = 'a5'`*\
*`--> @dm:dedup columns = 'ipaddress,hostname'`*\
*`--> *dm:filter * get id, hostname, ipaddress`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: RDA uses the dm function 'dm: dedup' to remove duplicate entries from the selected columns (IP Address, hostname) and prints the output for each dataset (or row) as shown in the following screenshot.
### Example 2:
Default dm: dedup functionality is captured in this example.
Step 1: Download 'incidents.csv' to the AIOps RDA environment as shown below from the local file system.
Step 2**:** Upload the file 'incidents.csv' to AIOps studio using file-browser (as shown below)
Step 3**:** Add a new empty pipeline with the name "**dm\_dedup\_example\_2**" as shown below and click the "Save" button (this step will create an empty pipeline and saves it to AIOps studio).
Step 4: Add the following pipeline commands into the empty pipeline text field that you have created in above Step 3.
You can copy the below code into your pipeline and execute that in your environment.\
*`##### This pipeline loads incidents.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from incidents.csv file`*\
*`##### into local dataset named 'incident-summary'.`*\
*`##### prints the data that was stored`*\
*`@files:loadfile filename = "incidents.csv"`*\
*`--> @dm:save name = 'incidents-summary'`*\
*`--> *dm:filter *`*
Step 5: Check the data from incidents.csv by executing the pipeline and verifying using inspect data as shown below (screenshot -1 & screenshot-2)
Step 6: Now, add the following additional pipeline code to use the dm: dedup function to the previously created pipeline from Step-4 as shown below (Edit and add the following pipeline code) and click verify to verify the pipeline code as shown below.
*`##### This pipeline loads incidents.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from incidents.csv file`*\
*`##### into local dataset named 'incident-summary'.`*\
*`##### prints the data that was stored`*\
*`@files:loadfile filename = "incidents.csv"`*\
*`--> @dm:save name = 'incidents-summary'`*\
*`--> *dm:filter *`* \
*`--> @dm:dedupe columns = 'Summary'`*
Step 7**:** Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 8: RDA uses the dm dedup to remove duplicate values from the requested column(s) and prints to output as shown below. \
Note: More columns can be selected as part of the dm: dedup function
Note: Total number of rows from incidents.csv was **436** before dedup function is run on the dataset. After dedup is run on the dataset, it reduces to **158** (as shown in the above screenshots). In this example, dedup column selected is 'Summary'. In addition, users can pick other columns from the dataset.
# Data Mapping cfxdm - dm:selectcolumns
**dm: selectcolumns:** This cfxdm tag allows the user to include or exclude columns using a regular expression format after retrieving the data from an extension's tag.
It is very useful when the user is dealing with many columns from an extension's tag and it simplifies columns select using either include or exclude or both options together
**dm:selectcolumns** syntax**:**
* **include (**optional**):** Specify a column or columns in regular expression format to include matched columns.
* **exclude (**optional**):** Specify a column or columns in regular expression format to exclude matched columns.
### Example 1: Select matched columns using the include option
Selectcolumns functionality is captured in this example.
Step 1: Create an empty **dm\_selectcolumns\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a set of records/dateset details coming from a vCenter`* \
*`##### environment. This dataset includes datastore details, Folder name, guest_hostname,`* \
*`##### guest_ip_address.`*\
\
*`###### This pipeline uses dm selectcolumns to select user provided column names`*\
*`##### and prints the output.`*\
\
*`@dm:empty`* \
*`--> @dm:addrow datastore = 'CFX-QA-Store-NFS-qnap' & Folder = '' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'test2.oia.cloudfabrix.com' & guest_ip_address = '10.95.134.17'`* \
\
*`--> @dm:addrow datastore = 'CFX-QA-Store-NFS-qnap' & Folder = 'User VMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'qa.oia.cloudfabrix.com' & guest_ip_address = '10.95.122.13'`*\
\
*`--> @dm:addrow datastore = 'ENG_ISOs,netapp-qa-nfs' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'Ubuntu Linux (64 bit)' & guest_hostname = 'ubuntuqa' & guest_ip_address = '10.95.102.172'`*\
\
*`--> @dm:addrow datastore = 'datastore1 (6)' & Folder = 'Ravi-Pisupati' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'elkstack' & guest_ip_address = '10.95.121.218'`*\
\
*`--> @dm:addrow datastore = 'datastore1-198' & Folder = '' & guest_full_name = 'Microsoft Windows Server' & guest_hostname = 'spdnode.adwinstack' & guest_ip_address = '10.95.132.4'`* \
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-01' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'localhost' & guest_ip_address = '10.95.103.115'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-01' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'localhost' & guest_ip_address = '10.95.103.115'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'democloudp' & guest_ip_address = '10.95.122.212'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'ravip-v201-platform' & guest_ip_address = '10.95.122.216'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'cfxautomatesvc' & guest_ip_address = '10.95.125.116'`*\
\
*`--> @dm:selectcolumns include = 'datastore.*|guest.*'`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below).
Step 5: RDA uses the dm selectcolumns function to select 'datastore and also guest\*' based columns as shown in the below screenshot.
### Example 2: Select matched columns using the include option and deselect columns using the 'exclude' option
The following pipeline captures this example.
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a set of records/dateset details coming from a vCenter`* \
*`##### environment. This dataset includes datastore details, Folder name, guest_hostname,`* \
*`##### guest_ip_address.`*\
\
*`###### This pipeline uses dm selectcolumns to select user provided column names`*\
*`##### using include, exclude options and prints the output.`*\
\
*`@dm:empty`* \
*`--> @dm:addrow datastore = 'CFX-QA-Store-NFS-qnap' & Folder = '' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'test2.oia.cloudfabrix.com' & guest_ip_address = '10.95.134.17'`* \
\
*`--> @dm:addrow datastore = 'CFX-QA-Store-NFS-qnap' & Folder = 'User VMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'qa.oia.cloudfabrix.com' & guest_ip_address = '10.95.122.13'`*\
\
*`--> @dm:addrow datastore = 'ENG_ISOs,netapp-qa-nfs' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'Ubuntu Linux (64 bit)' & guest_hostname = 'ubuntuqa' & guest_ip_address = '10.95.102.172'`*\
\
*`--> @dm:addrow datastore = 'datastore1 (6)' & Folder = 'Ravi-Pisupati' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'elkstack' & guest_ip_address = '10.95.121.218'`*\
\
*`--> @dm:addrow datastore = 'datastore1-198' & Folder = '' & guest_full_name = 'Microsoft Windows Server' & guest_hostname = 'spdnode.adwinstack' & guest_ip_address = '10.95.132.4'`* \
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-01' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'localhost' & guest_ip_address = '10.95.103.115'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-01' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'localhost' & guest_ip_address = '10.95.103.115'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'democloudp' & guest_ip_address = '10.95.122.212'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'ravip-v201-platform' & guest_ip_address = '10.95.122.216'`*\
\
*`--> @dm:addrow datastore = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'cfxautomatesvc' & guest_ip_address = '10.95.125.116'`*\
\
*`--> @dm:selectcolumns include = 'datastore.*|guest.*' & exclude = 'guest_full.*'`*
RDA uses the dm selectcolumns function to select 'datastore and also guest\*' based columns and excludes 'guest\_full \_name' as shown in the below screenshot
# Data Mapping cfxdm - dm:fixcolumns
**dm: fixcolumns:** This dm function allows the user to remove the special characters like @,. (dot) etc from a column name. If there is a special character in between a column name, it replaces it with an underscore (ex: First. Last to First\_Last)
**dm: fixcolumns** syntax**:** It doesn't require any arguments. Just ingest the data into this tag/function using a pipe (-->)
### Example 1:
Default dm: fixcolumns functionality is captured in this example.
Step 1: Create an empty **dm\_fixcolumns\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a set of records/dateset details coming from a vCenter`* \
*`##### environment. This dataset includes datastore details, Folder name, guest_hostname,`* \
*`##### guest_ip_address.`*\
\
*`###### This pipeline uses dm fixcolumns to fix the column names that has special`* \
*`###### characters and replace with '_' character for other operations.`*\
\
*`@dm:empty`* \
*``--> @dm:addrow datastore.1 = 'CFX-QA-Store-NFS-qnap' & Folder = '' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'test2.oia.cloudfabrix.com' & guest_ip_address = '10.95.134.17' & `VM Name` = 'test1'``*\
\
*``--> @dm:addrow datastore.1 = 'CFX-QA-Store-NFS-qnap' & Folder = 'User VMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'qa.oia.cloudfabrix.com' & guest_ip_address = '10.95.122.13' & `VM Name` = 'test2'``*\
\
*``--> @dm:addrow datastore.1 = 'ENG_ISOs,netapp-qa-nfs' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'Ubuntu Linux (64 bit)' & guest_hostname = 'ubuntuqa' & guest_ip_address = '10.95.102.172' & `VM Name` = 'test3'``*\
\
*``--> @dm:addrow datastore.1 = 'datastore1 (6)' & Folder = 'Ravi-Pisupati' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'elkstack' & guest_ip_address = '10.95.121.218' & `VM Name` = 'test4'``*\
\
*`--> @dm:addrow datastore.1 = 'datastore1-198' & Folder = '' & guest_full_name = 'Microsoft Windows Server' & guest_hostname = 'spdnode.adwinstack' & guest_ip_address = '10.95.132.4'`* \
*``& `VM Name` = 'test5'``*\
\
*``--> @dm:addrow datastore.1 = 'netapp-dev-nfs-01' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'localhost' & guest_ip_address = '10.95.103.115' & `VM Name` = 'test6'``*\
\
*``--> @dm:addrow datastore.1 = 'netapp-dev-nfs-01' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'localhost' & guest_ip_address = '10.95.103.115' & `VM Name` = 'test7'``*\
\
*``--> @dm:addrow datastore.1 = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'democloudp' & guest_ip_address = '10.95.122.212' & `VM Name` = 'test8'``*\
\
*``--> @dm:addrow datastore.1 = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'ravip-v201-platform' & guest_ip_address = '10.95.122.216' & `VM Name` = 'test9'``*\
\
*``--> @dm:addrow datastore.1 = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'cfxautomatesvc' & guest_ip_address = '10.95.125.116' & `VM Name` = 'test10'``*\
\
*`--> @dm:fixcolumns`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below).
Step 5: RDA uses the dm fixcolumns function to replace special characters (dot, space, etc) with underscore character as shown in the following screenshot.
# Data Mapping cfxdm - dm:mergecolumns
Data merge from multiple columns to a single column
**dm:mergecolumns:** This cfxdm function allows the user to select multiple columns using include or exclude columns options using a regular expression format and merge them into a single target column.
**dm:mergecolumns** syntax**:**
* **include (**Mandatory**):** Specify a column or columns in regular expression format to include matched columns.
* **exclude (**Optional**):** Specify a column or columns in regular expression format to exclude matched columns.
* **to** (Mandatory): Target column for merged data from selected one or more columns.
{% hint style="info" %}
**Note:** After merging multiple columns into a single column, it will remove source columns from the final output.
{% endhint %}
### Example 1:
Step 1: Create an empty **dm\_mergecolumns\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a set of records/dateset details coming from a netflow`* \
*`##### environment.`* \
\
*`##### This dataset includes flow.client_addr, flow.server_addr, flow.service_port as`* \
*`##### source columns. This pipeline uses dm mergecolumns to merge the source columns`*\
*`##### to a single target column`*\
\
*`@dm:empty`* \
*`--> @dm:addrow flow.client_addr = '10.95.133.42' & flow.server_addr = '10.95.133.40' & flow.service_port = '9092'`* \
*`--> @dm:addrow flow.client_addr = '10.95.122.170' & flow.server_addr = '10.95.122.169' & flow.service_port = '9092'`*\
*`--> @dm:addrow flow.client_addr = '10.95.122.170' & flow.server_addr = '10.95.122.169' & flow.service_port = '9092'`* \
*`--> @dm:addrow flow.client_addr = '10.95.122.170' & flow.server_addr = '10.95.122.169' & flow.service_port = '9092'`* \
*`--> @dm:addrow flow.client_addr = '10.95.122.205' & flow.server_addr = '10.95.122.212' & flow.service_port = '9300'`* \
*`--> @dm:addrow flow.client_addr = '10.95.117.35' & flow.server_addr = '10.95.117/37' & flow.service_port = '686'`* \
*`--> @dm:addrow flow.client_addr = '10.95.122.105' & flow.server_addr = '10.95.122.212' & flow.service_port = '443'`* \
*`--> @dm:addrow flow.client_addr = '10.95.122.105' & flow.server_addr = '10.95.122.212' & flow.service_port = '9300'`* \
*`--> @dm:addrow flow.client_addr = '10.95.122.121' & flow.server_addr = '10.95.122.212' & flow.service_port = '9092'`*\
*`--> @dm:addrow flow.client_addr = '10.95.122.109' & flow.server_addr = '10.95.122.212' & flow.service_port = '8443'`*\
\
*`--> @dm:mergecolumns include = 'flow.client_addr|flow.server_addr|flow.service_port' & to = 'flow.unique_id'`*
Step 3: Click verify button to make sure syntax and pipeline code is correct (as shown below)
Step 4: Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below).
Step 5: RDA uses the dm mergecolumns function to merge the specified columns into target column and prints the resultant output as shown in the following screenshot.
dm: merge functionality will be very useful when uniqueness is needed from within the dataset by combining couple of columns into one column as a key.
# Data Mapping cfxdm - dm:describe
Show column statistics
**dm:describe:** This cfxdm tag allows the user to select one or more columns and provides below summarized statistics on the values within them.
* **count:** Total count of the values within a column
* **unique:** Total unique count of the values within a column
* **top:** The top value out of all unique values which are repeated compared to others
* **freq:** The number of times the top unique is repeated
* **mean:** The mean out of all of the values (applies to rows that only has numeric values)
* **std:** The standard deviation out of all of the values (applies to rows that only has numeric values)
* **min:** The minimum value out of all of the values (applies to rows that only has numeric values)
* **25%:** 25% percentile out of all of the values (applies to rows that only has numeric values)
* **50%:** 50% percentile out of all of the values (applies to rows that only has numeric values)
* **75%:** 75% percentile out of all of the values (applies to rows that only has numeric values)
* **max:** The maximum value out of all of the values (applies to rows that only has numeric values)
**dm: describe** syntax**:**
* **columns (**optional**)**. Specify a column or columns (comma separated) on which '**describe**' to be applied.
This section explains how users can use a CSV file loaded into a dataset. This saved dataset will be used to explain how the dm: describe function to display the stats of the dataset.
{% hint style="info" %}
Download the [incidents.csv](https://macaw-amer.s3.amazonaws.com/rda/data/incidents.csv) file to the local machine as shown below using a standard web browser
{% endhint %}
### Example 1:
Step 1: Download 'incidents.csv' to the AIOps RDA environment as shown below from the local file system.
Step 2**:** Upload the file 'incidents.csv' to AIOps studio using file-browser (as shown below)
Step 3**:** Add a new empty pipeline with the name "**dm\_describe\_example\_1**" as shown below and click the "Save" button (this step will create an empty pipeline and saves it to AIOps studio).
Step 4: Add the following pipeline commands into the empty pipeline text field that you have created in above Step 3.
You can copy the below code into your pipeline and execute that in your environment.\
*`##### This pipeline loads incidents.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from incidents.csv file`*\
*`##### into local dataset named 'incident-summary'.`*\
*`##### prints the data that was stored`*\
*`@files:loadfile filename = "incidents.csv"`*\
*`--> @dm:save name = 'incidents-summary'`*\
*`--> *dm:filter *`*
Step 5: Check the data from incidents.csv by executing the pipeline and verifying using inspect data as shown below (screenshot -1 & screenshot-2)
Step 6: Now, add the following additional pipeline code to use the dm: describe function to the previously created pipeline from Step-4 as shown below (Edit and add the following pipeline code) and click verify to verify the pipeline code as shown below.
*`##### This pipeline loads incidents.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from incidents.csv file`*\
*`##### into local dataset named 'incident-summary'.`*\
*`##### prints the data that was stored`*\
*`@files:loadfile filename = "incidents.csv"`*\
*`--> @dm:save name = 'incidents-summary'`*\
*`--> *dm:filter *`* \
*`--> @dm:describe`*
Step 7**:** Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 8: RDA uses the dm describe function to describe the statistics of the selected 'incidents.csv' dataset that was loaded earlier (and saved) as shown in the following screenshot.
### Example 2:
Now, add the following additional pipeline code to use the dm: describe function to the previously created pipeline from Step-4 as shown below (Edit and add the following pipeline code) and click verify to verify the pipeline code as shown below.
*`##### This pipeline loads incidents.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from incidents.csv file`*\
*`##### into local dataset named 'incident-summary'.`*\
*`##### prints the data that was stored`*\
*`@files:loadfile filename = "incidents.csv"`*\
*`--> @dm:save name = 'incidents-summary'`*\
*`--> *dm:filter *`* \
*`--> @dm:describe columns = 'Summary,Source'`*
Once the above pipeline code is added and executed, stats are provided to selected two columns as shown below screenshot.
# Data Mapping cfxdm - dm:save
Save input data into a named dataset
**dm:save:** This dm function allows the user to save the retrieved data from an extension's tag as a named dataset for later consumption.
**dm: save** syntax**:** Below arguments are supported.
* **name** (mandatory): Name (unique) of the dataset to which data needs to be saved.
* **publish** (optional): Name of the tag to ingest or publish into cfxDimensions platform. \
\
*Note: It can be used only with cfxDimensions platform configuration.*
{% hint style="info" %}
This extension tag is typically used along with **dm: recall**
{% endhint %}
{% hint style="info" %}
To list the named datasets, use **dm: savedlist**
{% endhint %}
This section explains how users can use a CSV file loaded into a dataset. This saved dataset will be used to explain how the dm: head function can be used to check the head of the stored dataset.
{% hint style="info" %}
Download the [incidents.csv](https://macaw-amer.s3.amazonaws.com/rda/data/incidents.csv) file to the local machine as shown below using a standard web browser.
{% endhint %}
### Example 1:
Default dm: save functionality is captured in this example.
Step 1: Download 'incidents.csv' to the AIOps RDA environment as shown below from the local file system.
Step 2**:** Upload the file 'incidents.csv' to AIOps studio using file-browser (as shown below)
Step 3**:** Add a new empty pipeline with the name "**dm\_save\_example\_1**" as shown below and click the "Save" button (this step will create an empty pipeline and saves it to AIOps studio).
Step 4: Add the following pipeline commands into the empty pipeline text field that you have created in above Step 3.
You can copy the below code into your pipeline and execute that in your environment.\
*`##### This pipeline loads incidents.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from incidents.csv file`*\
*`##### into local dataset named 'incident-summary'.`*\
*`##### prints the data that was stored`*\
*`@files:loadfile filename = "incidents.csv"`*\
*`--> @dm:save name = 'incidents-summary'`*\
*`--> *dm:filter *`*
Step 5: Check the data from incidents.csv by executing the pipeline and verifying using inspect data as shown below (screenshot -1 & screenshot-2)
Step 5: Use dm: savedlist to check the logical datasets that were used to store the data using dm: save function as shown in the following screenshot.
You can copy the below code into your pipeline and execute that in your environment to verify that incidents-summary dataset is stored using the pipeline \
\
*`##### This pipeline loads incidents.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from incidents.csv file`*\
*`##### into local dataset named 'incident-summary'.`*\
*`##### prints the data that was stored`*\
*`@files:loadfile filename = "incidents.csv"`*\
*`--> @dm:save name = 'incidents-summary'`*\
\
*`--> @c:new-block`*\
*`--> *dm:savedlist`*
Step 6**:** Click execute button and execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 7: Verify that the incidents-summary dataset is part of the dm:savedlist call as shown in the below screenshot.
# Data Mapping cfxdm - dm:savedlist
List named datasets
**dm: savedlist:** This dm function allows the user to list the saved named datasets.
**dm: savedlist** syntax**:** It doesn't require any arguments
{% hint style="info" %}
This tag is typically used along with the **dm: save**
{% endhint %}
Use **dm: savedlist** tag to list saved named datasets. Use [**dm: save**](https://app.gitbook.com/@cloudfabrix/s/docs/rda/rda-userguide/rda-aiops-studio/examples-jupyter/data-mapping-cfxdm-dm-save) function to save dataset for incidents.csv as explained in dm: save section. Once the dataset is saved using dm: save, dm: savedlist is used to list the saved datasets.
Step 1: Use dm: savedlist to check the logical datasets that were used to store the data using dm: save function as shown in the following screenshot.
Create a pipeline "**dm\_saved\_example\_1**" and copy the below code into your pipeline and execute that in your environment. \
\
*`##### This pipeline loads incidents.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from incidents.csv file`*\
*`##### into local dataset named 'incident-summary'.`*\
*`##### prints the data that was stored`*\
*`@files:loadfile filename = "incidents.csv"`*\
*`--> @dm:save name = 'incidents-summary'`*\
\
*`--> @c:new-block`*\
*`--> *dm:savedlist`*
Step 2: Verify that the incidents-summary dataset is part of the dm:savedlist call as shown in the below screenshot.
# Data Mapping cfxdm - dm:recall
Recall or retrieve the data from saved named dataset
**dm:recall:** This cfxdm function allows the user to recall or retrieve the data from a saved (named) dataset.
**dm:recall** syntax**:**
* **name** (mandatory)**:** Name of the dataset to recall - **name = '\'**
* **cache** (optional)**:** Cache the result for future recalls. 'yes' or 'no' - **cache = '\' (**default is 'no**').**
* **cache\_refresh\_seconds** (optional)**: cache\_refresh\_seconds = '\' (**default 120 seconds**).** Refresh the cache (if new update available) after specified seconds
* **return\_empty** (optional): Return an empty dataframe if an error occurs loading the dataset - **return\_empty = '\' (**default is 'no**').**
* **empty\_df\_columns** (optional): Comma separated list of columns for empty dataframe.
Use [**dm:savedlist**](/rda/rda-userguide/rda-data-management-cfxdm/cfxdm-dm-savedlist) tag to list saved datasets
### Example 1:
Default dm: recall functionality is captured in this example.
This section explains how users can use a CSV file loaded into a dataset. This saved dataset will be used to explain how the dm: head function can be used to check the head of the stored dataset.
{% hint style="info" %}
Download the [incidents.csv](https://macaw-amer.s3.amazonaws.com/rda/data/incidents.csv) file to the local machine as shown below using a standard web browser.
{% endhint %}
### Example 1:
Default dm: save functionality is captured in this example.
Step 1: Download 'incidents.csv' to the AIOps RDA environment as shown below from the local file system.
Step 2**:** Upload the file 'incidents.csv' to AIOps studio using file-browser (as shown below)
Step 3**:** Add a new empty pipeline with the name "**dm\_recall\_example\_1**" as shown below and click the "Save" button (this step will create an empty pipeline and saves it to AIOps studio).
Step 4: Add the following pipeline commands into the empty pipeline text field that you have created in above Step 3.
You can copy the below code into your pipeline and execute that in your environment.\
*`##### This pipeline loads incidents.csv file into AIOps Studio.`* \
*`##### AIOps studio stores the data loaded from incidents.csv file`*\
*`##### into local dataset named 'incident-summary'.`*\
\
*`###### Pipeline uses saved list as well as recall functions to print the saved dataset`*\
*`###### output`* \
\
*`@files:loadfile filename = "incidents.csv"`*\
*`--> @dm:save name = 'incidents-summary'`*\
\
*`--> @c:new-block`* \
*`--> *dm:savedlist`* \
*`--> @dm:recall name = 'incidents-summary'`*
Step 5**:** Click verify button to verify the pipeline. RDA will verify the pipeline without any errors (as shown below)
Step 5**:** Click execute button to execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 6: Verify that the incidents-summary dataset is part of the dm: savedlist and dm:recall call as shown in the below screenshot.
# Data Mapping cfxdm - dm:concat
Merge or append two or more named datasets
**dm: concat:** This cfxdm function allows the user to merge two or more named datasets.
**dm:concat** syntax**:**
* **names** (mandatory)**:** List of two or more named datasets, supports regex
{% hint style="info" %}
Please refer **to dm: save** and **dm: savedlist** functions on how to create and list named datasets.
{% endhint %}
Use **dm: savedlist** tag to list saved datasets
### Example 1:
Step 1: Create an empty **dm\_concat\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.
\
*`###### Pipeline created two datasets simulating two alerting systems (PRTG, vrOPS) and`*\
*`###### generates data and saves into dataset.`*\
*`###### Pipeline uses dm:concat function to concatenate two datasets that are stored in RDA`*\
*`##### Pipeline also uses, dm:save and dm:savedlist`* \
\
*`####### PRTG Alerts Sections`* \
*`@dm:empty`* \
*`--> @dm:addrow prtg_alert_id = 'PRTG111122' & description = 'VM Outage' & priority = 'P0'`*\
*`--> @dm:addrow prtg_alert_id = 'PRTG111123' & description = 'Increase CPU' & priority = 'P1'`* \
*`--> @dm:addrow prtg_alert_id = 'PRTG111124' & description = 'Disk resize' & priority = 'P1'`* \
*`--> @dm:addrow prtg_alert_id = 'PRTG111125' & description = 'App Install' & priority = 'P2'`* \
*`--> @dm:addrow prtg_alert_id = 'PRTG111126' & description = 'DB bounce' & priority = 'P0'`* \
*`--> @dm:save name = "prtg-alerts"`*\
\
*`####### vrOPS Alerts Sections`* \
*`--> @c:new-block`*\
*`--> @dm:empty`* \
*`--> @dm:addrow alert_id = 'vrOps22221' & description = 'Outage' & priority = 'P0'`* \
*`--> @dm:addrow alert_id = 'vrOps22222' & description = 'Increase CPU' & priority = 'P1'`* \
*`--> @dm:addrow alert_id = 'vrOps22223' & description = 'Disk resize' & priority = 'P1'`* \
*`--> @dm:addrow alert_id = 'vrOps22224' & description = 'App Install' & priority = 'P2'`* \
*`--> @dm:addrow alert_id = 'vrOps22225' & description = 'DB bounce' & priority = 'P0'`* \
*`--> @dm:save name = "vrops-alerts"`*\
\
*`--> @c:new-block`* \
*`--> @dm:concat names = 'prtg-alerts|vrops-alerts'`* \
*`--> dm:save name = 'consolidated-alerts'`*\
*`--> @c:new-block --> *dm:savedlist`*
Step 3**:** Click verify button to verify the pipeline. RDA will verify the pipeline without any errors (as shown below)
Step 5**:** Click execute button to execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 6: Verify that the incidents-summary dataset is part of the dm: savedlist and dm:recall call as shown in the below screenshot.
# Data Mapping cfxdm - dm:groupby
Group rows by selected columns
**dm: groupby:** This cfxdm function allows the user to group the data (by rows) based on selected columns using aggregate functions.
**dm: groupby** syntax**:**
* **columns** (mandatory)**:** Select one or more columns for grouping the data.
* **agg** (optional)**:**
* **count:** It is applied by default when 'agg' is not specified. Supported on any value types (numeric or non-numeric values)
* **min:** Supported on numeric values only
* **max:** Supported on numeric values only
* **sum:** Supported on numeric values only
### Example 1:
Step 1: Create an empty **dm\_groupby\_example\_1** using AIOps studio as shown in the below screenshot.
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.\
\
*`##### This pipeline creates a set of records/dateset details coming from a vCenter`* \
*`##### environment. This dataset includes datastore details, Folder name, guest_hostname,`* \
*`##### guest_ip_address.`*\
\
*`###### This pipeline uses dm groupby to view the data grouped by selective column name`*\
*`###### Example by Folder, datastore etc.`*\
\
*`@dm:empty`* \
*``--> @dm:addrow datastore.1 = 'CFX-QA-Store-NFS-qnap' & Folder = '' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'test2.oia.cloudfabrix.com' & guest_ip_address = '10.95.134.17' & `VM Name` = 'test1'``*\
\
*``--> @dm:addrow datastore.1 = 'CFX-QA-Store-NFS-qnap' & Folder = 'User VMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'qa.oia.cloudfabrix.com' & guest_ip_address = '10.95.122.13' & `VM Name` = 'test2'``*\
\
*``--> @dm:addrow datastore.1 = 'ENG_ISOs,netapp-qa-nfs' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'Ubuntu Linux (64 bit)' & guest_hostname = 'ubuntuqa' & guest_ip_address = '10.95.102.172' & `VM Name` = 'test3'``*\
\
*``--> @dm:addrow datastore.1 = 'datastore1 (6)' & Folder = 'Ravi-Pisupati' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'elkstack' & guest_ip_address = '10.95.121.218' & `VM Name` = 'test4'``*\
\
*`--> @dm:addrow datastore.1 = 'datastore1-198' & Folder = '' & guest_full_name = 'Microsoft Windows Server' & guest_hostname = 'spdnode.adwinstack' & guest_ip_address = '10.95.132.4'`* \
*``& `VM Name` = 'test5'``*\
\
*``--> @dm:addrow datastore.1 = 'netapp-dev-nfs-01' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'localhost' & guest_ip_address = '10.95.103.115' & `VM Name` = 'test6'``*\
\
*``--> @dm:addrow datastore.1 = 'netapp-dev-nfs-01' & Folder = 'User VMs,Ravi-Pisupati,TestVMs' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'localhost' & guest_ip_address = '10.95.103.115' & `VM Name` = 'test7'``*\
\
*``--> @dm:addrow datastore.1 = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'democloudp' & guest_ip_address = '10.95.122.212' & `VM Name` = 'test8'``*\
\
*``--> @dm:addrow datastore.1 = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'ravip-v201-platform' & guest_ip_address = '10.95.122.216' & `VM Name` = 'test9'``*\
\
*``--> @dm:addrow datastore.1 = 'netapp-dev-nfs-02' & Folder = 'CFX-Drama,Capri-Demo' & guest_full_name = 'CentOS 4/5/6/7 (64 bit)' & guest_hostname = 'cfxautomatesvc' & guest_ip_address = '10.95.125.116' & `VM Name` = 'test10'``*\
\
*`--> @dm:groupby columns = 'Folder'`*
Step 3**:** Click verify button to verify the pipeline. RDA will verify the pipeline without any errors (as shown below)
Step 4**:** Click execute button to execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: Verify that the output data is grouped based on the selected column (e.g. 'Folder' column from the dataset created) as shown below screenshot.
This function is useful when users want to group by column/columns. In addition, is useful for any numeric values.
{% hint style="info" %}
When an agg function '**sum**' is used, it automatically applies to all eligible columns (numerical values).
{% endhint %}
# Data Mapping cfxdm - dm:to\_type
Set data type on columns
**dm:to\_type:** It allows to change or set the data type to string or integer (numeric) or float for specified columns.
@param '**columns**' (mandatory, comma separated list of column names), @param '**type**' (mandatory, supported options are: str / int / float)
**dm:to\_type:** Syntax
* **columns** (mandatory)**:** Comma-separated list of column names
* **type** (mandatory)**:** Supported options are: **str** (string or text) / **int** (integer or numeric) / **float**
### Example 1:
Step 1: Create an empty **dm\_to\_type\_example\_1** using AIOps studio as shown in the below screenshot
*`##### This pipeline creates a set of records/dateset details with different datatypes`*\
*`##### String and integer.`*\
\
*`###### This pipeline uses dm to_type to convert basic string data type to int datatype`*\
\
~~*`--> @dm:empty`*~~ \
~~*`--> @dm:addrow id = 'a1' & x = '10.0' & y = '3.11' & z = 9`*~~ \
~~*`--> @dm:addrow id = 'a2' & x = '1.0' & y = '13.0' & z = 4`*~~ \
~~*`--> @dm:addrow id = 'a3' & x = '7.0' & y = '3.0' & z = 3`*~~ \
~~*`--> @dm:addrow id = 'a4' & x = '4.0' & y = '-1.4' & z = 9`*~~\
~~*`--> @dm:to_type columns ='x,y' & type = 'int'`*~~
Step 3**:** Click verify button to verify the pipeline. RDA will verify the pipeline without any errors (as shown below)
Step 4**:** Click execute button to execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: Verify that the output data is enriched with the requested column names and print the output as shown in the below screenshot.
As shown in the above pipeline, this function is used to convert string to int (or float) datatype needed for any integer manipulation, etc.
# Data Mapping cfxdm - dm:enrich
Enrich the data using dictionaries
**dm:enrich:** This cfxdm function allows the user to enrich an existing dataset by looking into additional datasets or dictionaries and brings in additional enriched information from them as per the user's selection and requirement.
**dm: enrich** syntax**:**
* **dict** (mandatory)**:** Dictionary name (named dataset) which has additional enrichment data.
* **src\_key\_cols** (mandatory)**:** Named dataset's (source) key columns, comma separated.
* **dict\_key\_cols** (mandatory)**:** Dictionary name's (named dataset) key columns, comma separated.
* **enrich\_cols** (mandatory)**:** Enriched column names from Dictionary (named dataset) selected under '**dict**' option, comma-separated.
{% hint style="info" %}
The number of selected columns (count), for both **src\_key\_cols** & **dict\_key\_cols** options should be same.
i.e. if two columns are specified in **src\_key\_cols,** make sure two columns are specified in **dict\_key\_cols** too**.**
{% endhint %}
### Example 1:
Step 1: Create an empty **dm\_enrich\_example\_1** using AIOps studio as shown in the below screenshot
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.
\
*`###### Pipeline created two datasets simulating two datasets app-processes-list and`* \
*`###### apps-services-list from windows environment/inventory.`*\
\
*`###### Pipeline uses dm:enrich function to enrich two datasets that are stored in RDA`*\
*`##### Pipeline also uses, dm:recall`* \
\
*`####### app-processes-list dataset`*\
*`@dm:empty`* \
*`--> @dm:addrow ip_address = '10.95.108.100' & process_name = 'System Idle Process' & pid = 0`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & process_name = 'svchost.exe' & pid = 1020`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & process_name = 'svchost.exe' & pid = 1020`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & process_name = 'silsvc.exe' & pid = 1064`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & process_name = 'smbhash.exe' & pid = 1152`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & process_name = 'svchost.exe' & pid = 1172`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & process_name = 'taskhostex.exe' & pid = 1432`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & process_name = 'Microsoft AD' & pid = 1500`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & process_name = 'svchost.exe' & pid = 1520`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & process_name = 'dwm.exe' & pid = 1524`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & process_name = 'dfsrs.exe' & pid = 1556`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & process_name = 'svchost.exe' & pid = 776`*\
*`--> @dm:save name = 'app-processes-list'`*\
\
\
*`@dm:empty`* \
*`--> @dm:addrow ip_address = '10.95.108.100' & service_name = 'ADWS' & State = 'Running' & pid = 1500`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & service_name = 'AppHostSvc' & State = 'Running' & pid = 1524`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & service_name = 'BFE' & State = 'Running' & pid = 1172`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & service_name = 'BITS' & State = 'Running' & pid = 1020`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & service_name = 'BrokerInfrastructure' & State = 'Running' & pid = 776`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & service_name = 'ComsysApp' & State = 'Running' & pid = 3584`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & service_name = 'CertPropsSvc' & State = 'Running' & pid = 1020`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & service_name = 'CryptSvc' & State = 'Running' & pid = 820`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & service_name = 'DFSR' & State = 'Running' & pid = 1556`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & service_name = 'DNS' & State = 'Running' & pid = 1624`*\
*`--> @dm:addrow ip_address = '10.95.108.100' & service_name = 'DPS' & State = 'Running' & pid = 1172`*\
*`--> @dm:save name = 'app-services-list'`*\
\
\
*`--> @c:new-block`* \
*`--> @dm:recall name = 'app-services-list'`*\
*`--> @dm:enrich dict = 'app-processes-list' & src_key_cols = 'ip_address,pid' & dict_key_cols = 'ip_address,pid' & enrich_cols = 'process_name'`*
Step 3**:** Click verify button to verify the pipeline. RDA will verify the pipeline without any errors (as shown below)
Step 4**:** Click execute button to execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: Verify that the output data is enriched with the requested column names and print the output as shown in the below screenshot.
As shown in the above screenshot, 'app-services-list' is enriched using additional dictionary 'app-processes-list' using key columns that match from both datasets. In addition, the pipeline enriches the output with process\_name. In this example, only a single column was enriched from the process list. But, the same enrichment process can be extended to other columns based on the other datasets that users want to enrich.
# Data Mapping cfxdm - dm:dns\_ip\_to\_name
DNS Name resolution from IP Address to FQDN
**dm:dns\_ip\_to\_name:** It allows to resolve the IP addresses listed in a column to FQDN names into another column.
**dm:dns\_ip\_to\_name:** Syntax
* **from\_cols** (mandatory)**:** Comma-separated list of column names which has IP Address values
* **to\_cols** (mandatory)**:** Comma-separated list of column names to store resolved DNS Names (FQDN).
* **keep\_value** (optional): If it is set to **'yes',** it stores the original value, else it stores the 'null' value. By default, it is set to '**no**'
* **num\_threads** (optional): Number of DNS lookup threads. Must be in the range of 1 to 20, Default is set to 5.
{% hint style="info" %}
This function uses the DNS servers configured on the host os where **RDA** is installed and running for IP Address to DNS or FQDN name resolution.
{% endhint %}
{% hint style="info" %}
It requires an input dataset or a tag that has one or more columns with IP Address values.
{% endhint %}
### Example 1:
Step 1: Create an empty **dm\_dns\_ip\_to\_name\_example\_1** using AIOps studio as shown in the below screenshot
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.
\
*`###### Pipeline created a dataset with ip_address, process_name and pid (runtime`* \
*`###### information on a windows box.`*\
*`###### Pipeline uses dm:dns_ip_to_name function to resolve name using DNS server using`*\
*`###### RDA macahine where it is installed`*\
\
*`@dm:empty`* \
*`--> @dm:addrow ip_address = '10.95.122.221' & process_name = 'System Process' & pid = 0`*\
*`--> @dm:addrow ip_address = '10.95.122.221' & process_name = 'svchost.exe' & pid = 1020`*\
*`--> @dm:addrow ip_address = '10.95.122.221' & process_name = 'svchost.exe' & pid = 1020`*\
*`--> @dm:addrow ip_address = '10.95.122.221' & process_name = 'silsvc.exe' & pid = 1064`*\
*`--> @dm:addrow ip_address = '10.95.159.100' & process_name = 'smbhash.exe' & pid = 1152`*\
*`--> @dm:addrow ip_address = '10.95.122.221' & process_name = 'svchost.exe' & pid = 1172`*\
*`--> @dm:dns_ip_to_name from_cols = 'ip_address' & to_cols = 'fqdn' & keep_value = 'yes'`*
Step 3**:** Click verify button to verify the pipeline. RDA will verify the pipeline without any errors (as shown below)
Step 4**:** Click execute button to execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: Verify that the ip\_address field as part of the dataset is resolved and print the FQDN (full qualified domain name) using DNS resolution as shown in the below screenshot.
# Data Mapping cfxdm - dm:dns\_name\_to\_ip
DNS Name resolution from FQDN to IP Address
**dm:dns\_name\_to\_ip:** It allows to resolve the FQDN names to IP addresses listed in a column into another column.
**dm:dns\_name\_to\_ip:** Syntax
* **from\_cols** (mandatory)**:** Comma-separated list of column names which has DNS Name (FQDN) values
* **to\_cols** (mandatory)**:** Comma-separated list of column names to store resolved DNS Name (FQDN) names into IP Address.
* **keep\_value** (optional): If it is set to **'yes',** it stores the original value, else it stores the 'null' value. By default, it is set to '**no**'
* **num\_threads** (optional): Number of DNS lookup threads. Must be in the range of 1 to 20, Default is set to 5.
{% hint style="info" %}
This tag uses the DNS servers configured on the host os where **RDA** is installed and running for IP Address to DNS or FQDN name resolution.
{% endhint %}
{% hint style="info" %}
It requires an input dataset or a tag that has one or more columns with DNS Name (FQDN) values.
{% endhint %}
### Example 1:
Step 1: Create an empty **dm\_dns\_name\_to\_ip\_example\_1** using AIOps studio as shown in the below screenshot
Step 2: Add the following pipeline code/commands into the above-created pipeline as shown in the below screenshot:
You can copy the below code into your pipeline and execute that in your environment.
\
*`###### Pipeline creates a dataset with server fqdn and pid runtime`* \
*`###### information on a windows box.`*\
*`###### Pipeline uses dm:dns_name_to_ip function to resolve FQDN DNS name to ipaddress`*\
*`###### dm function uses DNS server from RDA machine where it is installed`*\
\
*`@dm:empty`* \
*`--> @dm:addrow address = 'ec-test1.engr.cloudfabrix.com' & pid = 0`*\
*`--> @dm:addrow address = 'dns-cfx-dc1.engr.cloudfabrix.com' & pid = 1020`*\
*`--> @dm:addrow address = 'cfx-services-dep-01.engr.cloudfabrix.com' & pid = 1020`*\
*`--> @dm:addrow address = 'cfxregistry.cloudfabrix.com' & pid = 1064`*\
*`--> @dm:dns_name_to_ip from_cols = 'address' & to_cols = 'ip_address' & keep_value = 'yes'`*
Step 3**:** Click verify button to verify the pipeline. RDA will verify the pipeline without any errors (as shown below)
Step 4**:** Click execute button to execute the pipeline. RDA will execute the pipeline without any errors (as shown below)
Step 5: Verify that the FQDN field as part of the dataset is resolved and print the IP Address using DNS resolution as shown in the below screenshot.
# AIOps Studio - Datasource Examples
# Elasticsearch (v1)
Elascticsearch integration with AIOps/RDA.
## Introduction
This section explains how to add Elasticsearch data source, ingest data into Elasticsearch and query the data using AIOps/RDA environment.
## Adding Elasticsearch as Datasource in 'RDA':
RDA's user interface is used to configure Elasticsearch data source.
**Step 1: Accessing RDA UI**
Login into RDA's user interface using a browser.
**https\://\:9998**
Under '**Notebook**', click on '**CFXDX Python 3**' box
**Step 2: Adding Elasticsearch data source instance to RDA/AIOps**
In the '**Notebook**' command box, type **`botadmin()`** and **`alt (or option) + Enter`** to open the data source administration menu.
Click on the '**Add**' menu and under **Type** drop-down, select **`elasticsearch`**
* **Type:** Datasource/Extension type. In this context, it is '**elasticsearch**'
* **name**: Datasource/Extension label which should be unique within the RDA
* **Hostname:** Elasticsearch IP Address or FQDN/DNS name
* **Username**: User account that was created with 'read-only' permissions
* **Password**: User account's password
Click on '**Check Connectivity**' to verify the network access and credentials validity from RDA to Elasticsearch instance. Once it is validated, click on the '**Add**' button to add Elasticsearch as the data source
**Step 3: Adding tag definition in RDA and associate with Elasticsearch index**
Once the user completes **Step 2** and checks/validates connectivity from RDA to elasticsearch, the user can now add/define a tag in RDA which maps to elasticsearch index that was created earlier.
In the '**Notebook**' command box, type **`botadmin()`** and **`alt (or option) + Enter`** to open the data source administration menu.
Click on the '**Edit**' menu and under **Type** drop-down, select **'es/elasticsearch'** item that was created in step 2 (as shown in the below screenshot).
Note: In the above RDA tags definition, RDA keeps track of tag (rda-to-elasticsearch) to that of elasticsearch index (rda\_to\_elasticsearch\_idx with unique id as idx).
The code snippet is captured in the below code block.
```
- tag: rda-to-elasticsearch
index: rda_to_elasticsearch_idx
update:
index: rda_to_elasticsearch_idx
ids:
- idx
```
**Note: Before performing step 3, make sure elasticsearch index (rda\_to\_elasticsearch\_idx) has been created ahead in elasticsearch instance and verified using standard tools (e.g. curl or postman)**
**Step 4: Adding data using RDA and storing in Elasticsearch using the mapping that was created**
Create a pipeline "***rda\_to\_elasticsearch\_example\_1***" and copy the below code into your pipeline and perform the rest of the steps in your environment. \
\
*`##### This pipeline creates couple of user names and ids using RDA/AIOps Studio.`* \
*`##### RDA uses the mapping that was created and stores the records into elasticsearch`*\
\
*`--> @dm:empty`* \
*`--> @dm:addrow idx = 1 & name = 'David' & lastname = 'Eiger' & email = 'deiger@hello.com'`*\
*`--> @dm:addrow idx = 2 & name = 'Emma' & lastname = 'Edge' & email = 'eedge@hello.com'`*\
*`--> @dm:addrow idx = 3 & name = 'John' & lastname = 'Seagul' & email = 'jseagul@hello.com'`*\
*`--> @dm:addrow idx = 4 & name = 'Peter' & lastname = 'Samuel' & email = 'psamuel@hello.com'`*\
*`--> @dm:addrow idx = 5 & name = 'Sean' & lastname = 'Taylor' & email = 'staylor@hello.com'`*\
*`--> #es:rda-to-elasticsearch`*
**Step 5: Verify the above-added pipeline using AIOps/RDA by selecting the 'Verify' button as shown in the below screenshot**
**Step 6: Execute the pipeline by selecting the 'Execute' button as shown in the below screenshot**
**Step 7: Verify the data stored in Elasticsearch using RDA (and or using curl command).**
**Method A -- Using Curl command**
Step 1: Log in to the machine where Elasticsearch instance is running using putty or any other SSH tool
```
bash# ssh macaw@10.95.103.111
```
Step 2: Once you log in, run the following curl command to validate the data stored
```
curl -X GET 'http://localhost:9200/rda_to_elasticsearch_idx/_search'?pretty=true
```
The above curl command will return the data as pretty formatted JSON output as shown below.
**Method B -- Using RDA pipeline**
Step 1: Create a pipeline "***verify\_elasticsearch\_to\_rda\_data\_01***" and copy the below code into your pipeline and perform the rest of the steps in your environment.
*`##### This pipeline verifies the data stored via RDA pipeline`* \
*`#####`* \
*`--> @c:new-block`*\
*`--> #es:rda-to-elasticsearch`*
Step 2: Verify the above-created pipeline using RDA/AIOps studio as shown in the below screenshot
Step 3: Execute and verify the output of the data using RDA/AIOps studio as shown in the below screenshot
The above example walks through Elasticsearch integration with RDA using a simple inline dataset creation of users (name, last name, etc.). In addition, the datasets can come from files and/or other data sources like MySQL, etc. Users can explore other data sources using the above-explained steps.
Also, in the above example, a single Elasticsearch index 'idx' has been used to walk through the use case. Users can also extend or add additional indices to make a unique index based on the use case in context.
# RDA - Data Management (cfxdm)
Data management & transformation
RDA provides comprehensive built-in data management & transformation capabilities through **cfxdm extension and tags.**
Below is the list of available tags within the cfxdm extension. (shown in the below screen).
In the following sections, we provide detailed usage with examples using which how each cfxdm extension tag helps the user to manipulate and transform the data and move the data between different datasources / extensions.
# cfxdm - dm:filter
Data filtering using CFX query language
**dm:filter:** This cfxdm tag allows the user to apply simple to complex filters on the retrieved data from a different extension / datasource using CFX query language.
This tag is useful when an extension / datasource does not support filtering the data at the source using cfx query language. So, the pre-requisite is to retrieve the data before using this (dm:filter) tag.
**dm:filter syntax:**
* **dm:filter *****\***** :** Use CFX query language to apply the filter. If you do not want to apply any query and to include all of the ingested data, use **'\*'** (without any quotes).
* **dm:filter *****\***** get *****COLUMN\_Name\_1, COLUMN\_Name\_2, COLUMN\_Name\_3***** .... :** Use this syntax to limit the scope to only the selected columns. It also maintains the selected column's specified order.
* **dm:filter *****\***** get *****COLUMN\_Name\_1***** as '*****New\_COLUMN\_Name\_1*****', *****COLUMN\_Name\_2***** as '*****New\_COLUMN\_Name\_2*****', *****COLUMN\_Name\_3***** as '*****New\_COLUMN\_Name\_3'*****... :** Use this syntax to limit the scope to only the selected columns and rename them with new column names. It also maintains the selected column's specified order.
{% hint style="info" %}
Please refer [**CFX query language**](/cfxql-cfx-query-language) section for detailed information on supported queries and their usage/syntax with examples.
{% endhint %}
In the below example, for reference, we are going to use **VMware vROps** as an extension to query the data and ingest it into **dm:filter** tag for applying the filtering capabilities using CFX query language.
Enter the below command to select **VMware vROps resources tag (@vrops:resources)**. (In this example, vrops name is used as a label to identify VMware vROps extension and it's tags. The label is defined while adding the extension in cfxdx configuration file or through UI.)
```
tag @vrops:resources
```
**Example 1:**
Get the data from VMware vROps resources tag (@vrops:resources) and modify the column names. Below shown columns (highlighted) are from vROps resource tag.
* **identifier** as Virtual\_Resource\_Identifier
* **name** as Virtual\_Resource\_Name
* **resource\_kind** as Virtual\_Resource\_Type
* **adapter\_kind** as Virtual\_Adapter\_Type
{% hint style="info" %}
**-->** symbol represents piping the data between two extension tags, the output of an extension tag becomes as an input to another extension tag. It is similar to using the pipe (I) command in Unix/Linux OS.
{% endhint %}
```
data --> dm:filter * get identifier as 'Virtual_Resource_Identifier', name as 'Virtual_Resource_Name', resource_kind as 'Virtual_Resource_Type', adapter_kind as 'Virtual_Adapter_Type'
```
**Example 2:**
Get the data from VMware vROps resources tag (@vrops:resources) and filter the data only to selective columns. Below shown (highlighted) are all of the columns from vROps resource tag. Out of them, limit the column selection in a specific order (ex: name, resource\_kind & identifier.... skip adapter\_kind)
* **identifier**
* **name**
* **resource\_kind**
* **adapter\_kind (**skip this column from the query**)**
```
data --> dm:filter * get name,resource_kind,identifier
```
**Example 3:**
Get the data from VMware vROps resources tag (@vrops:resources) and using CFX query language, filter the data that matches '**VirtualMachine**' under column '**resource\_kind**' and limit the column's scope only to name, resource\_kind & identifier.
```
data --> dm:filter resource_kind equals 'VirtualMachine' get name,resource_kind,identifier
```
**Example 4:**
Get the data from VMware vROps resources tag (@vrops:resources) and using CFX query language, filter the data that matches '**VirtualMachine**' or '**DistributedVirtualPortgroup**' or '**ResourcePool**' and include all of the available columns.
```
data --> dm:filter resource_kind contains 'VirtualMachine|DistributedVirtualPortgroup|ResourcePool'
```
{% hint style="info" %}
If the **column name** has any **special character** (ex: a dot or dash etc..), then it needs to be escaped with backquotes. For example, the column name is first.name, it should be specified as **\`first.name\`** (**backquote escape**)
{% endhint %}
## **Appendix:**
**ServiceNow Examples:**
In this section, we are going to use **ServiceNow** as an extension to query the data and ingest it into the **dm:filter** tag for applying the filtering capabilities using CFX query language.
Note: In order for the following examples to work, the main pre-requisite is to add your ServiceNow account details to CFXDX using the CFXDX configuration section.
**Example 1:**
Checking the connectivity to the ServiceNow instance via CFXDX.
```
> check snow
```
**Example 2:**
Following ServiceNow tags are available via CFXDX.
```
> tags snow
```
**Example 3:**
Get the data from the ServiceNow tag (#snow:incidents) and modify the column names. Below shown columns (highlighted) are from the ServiceNow incidents tag.
* **number**
* **short\_description**
* **state**
* **priority**
```
data --> dm:filter * get number as 'Ticket_Number', short_description as 'Description', state as 'State', priority as 'Priority'
```
**Example 4:**
Get the data from ServiceNow tag (#snow:incidents)) and using CFX query language, filter the data that matches '**email**' under column 'short\_description*'* and limit the column's scope only to Number, Short description. State.
```
data --> dm:filter short_description contains 'email' get number as 'Incident Numnber', short_description as 'Short Description', state as 'State'
```
**Example 5:**
Get the data from ServiceNow tag (#snow:incidents)) and using CFX query language, filter the data that matches '**email' or 'laptop' or 'VPN'** under column 'short\_description*'* and limit the column's scope only to Number, Short description. State.
```
data --> dm:filter short_description contains 'email|VPN|laptop' get number as 'Incident Numnber', short_description as 'Short Description', state as 'State'
```
# cfxdm - dm:map
**dm:map:** This cfxdm tag allows the user to manipulate or transform the columns and it's values.
Below are some of the operations you can perform using this tag.
* Copy the Column X along with it's values as is and create a new Column Y
* Transform values from a Column to something else using 'functions' based on user's requirement
**dm:map syntax:**
* **dm:map from = '*****COLUMN\_X*****' & to = '*****COLUMN\_Y*****'**
OR
* **dm:map attr = '*****COLUMN\_Y' &***** func *****= "\" & \***
{% hint style="info" %}
**Note:** Only single '**func**' is supported per **dm:map** tag usage.
{% endhint %}
In the below example, for a reference, we are going to use **VMware vROps** as an extension to query the data and ingest it into **dm:map** tag for data manipulation or transformation using cfxdm functions.
Enter the below command to select **VMware vROps VM Summary tag (**\*vrops:**vm\_summary**). (In this example, vrops name is used as a label to identify VMware vROps extension and it's tags. The label is defined while adding the extension in cfxdx configuration file or through UI.)
```
tag *vrops:vm_summary
```
**Example 1:** Clone the Column (copy)
Get the data from VMware vROps VM Summary tag (**\*vrops:vm\_summary**) which includes some of the below columns, **clone** the column '**guest\_full\_name**' to '**guest\_os\_type**'.
**Column Names:**
* guest\_hostname
* guest\_ip\_address
* **guest\_full\_name**
* datastore
* .....
```
data --> @dm:map from = 'guest_full_name' & to = 'guest_os_type'
```
**Example 2:** Clone the Column (copy) & Transform the data with function 'evaluate' (simple 'if' and 'else' condition using a python expression)
Get the data from VMware vROps VM Summary tag (**\*vrops:vm\_summary**) which includes some of the below columns, **clone** the column '**guest\_full\_name**' to '**guest\_os\_type**' and transform the values (data) within the '**guest\_os\_type**' based on the existing value.
If column **guest\_os\_type's** value contains '**Microsoft Windows**', transform it to '**Windows**', else keep the existing value.
**Column Names:**
* **guest\_full\_name** (from)
* **guest\_os\_type** (to)
**Column Values:** (some of them)
* Microsoft Windows Server 2008 R2 (64-bit) **--> Transform to 'Windows'**
* CentOS 4/5/6/7 (64-bit)
* Red Hat Enterprise Linux 7 (64-bit)
{% hint style="info" %}
In the below syntax, @dm:map is used twice, 1st step is to clone the column to new column and 2nd step is to transform the values on cloned column.
{% endhint %}
```
data --> @dm:map from = 'guest_full_name' & to = 'guest_type' --> @dm:map attr = 'guest_type' & func = 'evaluate' & expr = "'Windows' if 'Microsoft Windows' in guest_type else guest_type"
```
**Example 3:** Clone the Column (copy) & Transform the data with function '**evaluate**' (using multiple '**if**' and '**else**' conditions using a python expression)
Get the data from VMware vROps VM Summary tag (**\*vrops:vm\_summary**) which includes some of the below columns, **clone** the column '**guest\_full\_name**' to '**guest\_os\_type**' and transform the values (data) within the '**guest\_os\_type**' based on the existing value.
If column **guest\_os\_type's** value contains '**Microsoft Windows**', transform it to '**Windows**', else keep the existing value.
**Column Names:**
* **guest\_full\_name** (from)
* **guest\_os\_type** (to)
**Column Values:** (some of them)
* Microsoft Windows Server 2008 R2 (64-bit) **--> Transform to 'Windows'**
* CentOS 4/5/6/7 (64-bit) **--> Transform to 'Linux OS Opensource'**
* Red Hat Enterprise Linux 7 (64-bit) **--> Transform to 'Linux OS Commercial'**
```
data --> @dm:map from = 'guest_full_name' & to = 'guest_type' --> @dm:map attr = 'guest_type' & func = 'evaluate' & expr = "'Windows' if 'Microsoft Windows' in guest_type else 'Linux OS Opensource' if 'CentOS' in guest_type else 'Linux OS Commercial' if 'Red Hat Enterprise' in guest_type else 'Other OS'"
```
# cfxdm - dm:functions
Data manipulation and transformation
**dm:functions:** This cfxdm tag provides very comprehensive data manipulation & transformation functions and below are the details about them and their usage.
* **any\_non\_null:** Returns any non-null value from a list of input values, @param value is optional, if not specified, returns None when none of the listed values meet the criteria. Input must be a list (else treated as a single item list).
* **concat:** Adds prefix and suffix to the specified string, @param prefix type is string (optional). @param suffix type is a string (optional). Input must be a string. If the input is null, it is treated as ' '
* **datetime:** Parses input string and converts into an epoch milliseconds format number. Input must be a string. @param tzmap: type dict (optional). Dictionary of timezone mappings from custom/local to standard timezones. @param expr: type string (optional). A custom timestamp format with UTC/local timezone.
* **evaluate:** Given an expression evaluates the expression string. If performed on the dataframe row, it evaluates by passing the row as a dictionary. If performed on a single value, it expects an additional argument 'key' to be used in the expression. @param expr: The expression to evaluate. @param key: An optional 'key' if evaluated on a single value instead of a dictionary.
* **fixed:** Returns a fixed value specified by the 'value' parameter. @param value Type can be string or number. Input can be of any type.
* **formDecode:** Decodes input string to remove any URL encoded values. Requires no parameters. Input must be a string.
* **highest:** Returns highest non-null value from the list of integer values. @param default (optional), Type int. If provided none of the input values are non-null, returns default Input must be a number or list of numbers.
* **hours\_between:** Number of hours between two datetime strings. If only one specified, compare diff between now and that timestamp.
* **join:** Joins input list using an optional separator. @param **sep** (optional), default value is ' ' . Input is expected to be a list. If the input is not a list, it returns the value without joining.
* **jsonDecode:** Decodes input string into JSON object. Requires no parameters. Input must be a string.
* **lower**: Converts to lowercase text, requires no parameters. Input must be a string.
* **lowest**: Returns lowest non-null value from the list of int values. @param default (optional), Type int. If provided none of the input values are non-null, returns default, Input must be a number or list of numbers.
* **map\_values**: Maps input value using the specified name value dictionary. If no values match and ""\*"" key is provided, it returns the ""\*"" key's values, or else the original value will be maintained. Input must be a string.
* **match**: Matches a regular expression and extracts a specific value (if matched). @param expr Type string. Regular expression @param flags List of optional flags (A I M L S X) Input must be a string.
* **minutes\_between:** Number of minutes between two date-time strings. If only one is specified, it compares the difference between it and the current timestamp.
* **replace**: Replaces old value with new value in the input string @param oldvalue, Type string. @param new value, Type string. Input must be a string.
* **seconds\_between**: Number of seconds between two datetime strings. If only one is specified, it compares the difference between it and the current timestamp.
* **slice**: Slices a string or an array using specified indices. @param from-index Type int. Default value 0 @param to-index Type int. The default value is None. Input can be a string or a list. If neither, it converts input to a string.
* **split**: Splits the input using specified 'sep' separator. @param sep Type string. Optional. Default any whitespace characters. Input must be a string.
* **strip**: Strips white spaces from both sides of a string, Requires no parameters. Input must be a string.
* **timediff**:
* **to\_numeric**: Convert input value into numeric
* **ts\_to\_datetimestr:** Processes input number with specified '**unit**' (s,ms,ns,excel\_date) and converts the value to **datetime** string specified by '**format**', default is ISO format. Input must be a float or int. @param '**unit' (**Type string), must be s,ms,ns,excel\_date, default is '**ms**' @param '**format' (**Type string), default is None (ISO format)
* **upper**: Converts to uppercase text Requires no parameters. Input must be a string.
* **valueRef**: Extracts a specific item from the input dictionary object. @param path A dot '.' delineated path to the element within the dictionary, Input must be dictionary object.
* **when\_null**: If the specified value is null, it uses the value as per 'value' param @param value Type can be string or number. Input can be of any type.
### Examples:
#### **Convert Milliseconds to Human readable date & timestamp**
**Functions:**
* **datetime**
* **ts\_to\_datetimestr**
In the below syntax example, '**Alert\_Time**' is a column name that has a timestamp value in milliseconds format.
```
dm:map attr = 'Alert_Time' & func = 'ts_to_datetimestr' & unit = 'ms'
```
From: (Time in Milliseconds)
To: (Time in Date & Time format)
{% hint style="info" %}
In the above example, **vrops-alerts** is a named dataset that was created out of the VMware vROps extension's alerts tag.
{% endhint %}
#### **Convert Human readable Date & Timestamp to Milliseconds**
```
dm:map attr = 'Alert_Time' & func = 'datetime'
```
From: (Time in Date & Time format)
To: (Time in Milliseconds)
{% hint style="info" %}
In the above example, **prtg-alerts** is a named dataset that was created out of the PRTG Monitor extension's alerts tag.
{% endhint %}
#### **Join values from two to more columns into one.**
**Function:**
* **join**
In the below syntax example, '**resource\_kind & adapter\_kind**' are columns and their values are joined together in a new column called '**resource\_and\_vendor**' using the function '**join**' (optional separator '-' is used in this example, when it is not specified, the default separator is space.
```
dm:map from = 'resource_kind,adapter_kind' & to = 'resource_and_vendor' & func = 'join' & sep = '-'
```
**Source Columns & their values:**
**Destination Column (after join)**
####
#### Arithmetic Operations
**Function:**
* **Evaluate**
{% hint style="info" %}
To perform arithmetic operations on columns, make sure the column's data type is set to '**Integer**' (Numeric) or '**Float**'. Additionally, column values should not contain **NULL** or **Empty** values.
{% endhint %}
**Multiplication**: In the below example, **vm\_cpu\_sockets** & **vm\_cpu\_cores** are columns which has numeric values, using '**evaluate**' function, multiplied 2 column values together and saving the result in new column call '**vm\_cpu\_total**'.
```
dm:map to ='vm_cpu_total' & func = 'evaluate' & expr = 'vm_cpu_sockets * vm_cpu_cores'
```
**Bytes to GB**: In the below example, **vm\_disk\_size\_bytes** is a column that has disk capacity in bytes, using the '**evaluate**' function, create a new column '**vm\_disk\_size\_gb'** and convert the bytes into GB.
```
dm:map from = 'vm_disk_size_bytes' & to ='vm_disk_size_gb' --> dm:map attr = 'vm_disk_size_gb' & func = 'evaluate' & expr = 'vm_disk_size_gb / 1024 / 1024 / 1024'
```
# cfxdm - dm:sort
Sort the data in ascending or descending order
**dm:sort:** This cfxdm tag allows the user to sort the data for a given column(s) in ascending or descending order.
**dm:sort syntax:** It supports the below arguments
* **columns (mandatory)**. It accepts one or more column names. If more than one column is specified, use comma as a separator.
* **order (optional)**. Supported values are '**ascending**' or '**descending**'. When not specified, default applied sort order is 'ascending'
**dm:sort columns = 'COLUMN\_A,COLUMN\_B,..'**
OR
**dm:sort columns = 'COLUMN\_A,COLUMN\_B,..' & order = 'ascending/descending'**
In the below example, for a reference, we are going to use **VMware vROps** as an extension to query the data and ingest it into **dm:sort** tag for applying the sort capability.
Enter the below command to select **VMware vROps VM Summary tag (**\*vrops:**vm\_summary**). (In this example, vrops name is used as a label to identify VMware vROps extension and it's tags. The label is defined while adding the extension in cfxdx configuration file or through UI.)
```
tag *vrops:vm_summary
```
**Example 1:** Sort the selected column. (**ascending** order)
Get the data from VMware vROps VM Summary tag (**\*vrops:vm\_summary**) which includes some of the below columns, sort the data by '**datastore**' column.
**Column Names:**
* guest\_hostname
* guest\_ip\_address
* guest\_full\_name
* **datastore**
* ...
Below command queries the data from vROps vm\_summary tag and sorts the data by column name 'datastore' in **ascending** order. (when '**order**' (optional) param is not specified, it sorts in ascending order by default).
```
data --> dm:sort columns = 'datastore'
```
**Example 2:** Sort the selected column. (**descending** order)
Get the data from VMware vROps VM Summary tag (**\*vrops:vm\_summary**) which includes some of the below columns, sort the data by '**datastore**' column.
**Column Names:**
* guest\_hostname
* guest\_ip\_address
* guest\_full\_name
* **datastore**
* ...
Below command queries the data from vROps vm\_summary tag and sorts the data by column name 'datastore' in **descending** order.
```
data --> dm:sort columns = 'datastore' & order = 'descending'
```
**Example 3:** Sort the selected multiple columns. (**descending** order)
Get the data from VMware vROps VM Summary tag (**\*vrops:vm\_summary**) which includes some of the below columns, sort the data by '**guest\_full\_name**' and '**datastore**' columns.
**Column Names:**
* guest\_hostname
* guest\_ip\_address
* **guest\_full\_name**
* **datastore**
* ...
Below command queries the data from vROps vm\_summary tag and sorts the data by both column names '**guest\_full\_name**' and '**datastore**' in **descending** order.
```
data --> dm:sort columns = 'datastore,guest_full_name' & order = 'descending'
```
# cfxdm - dm:head
Get top 'n' rows
**dm:head:** This cfxdm tag allows the user to fetch top 'n' rows from the queried data.
**dm:head** syntax**:**
* **n (optional)**. Specify number of top rows that need to be listed. When this argument is not specified, by default it retrieves top 10 rows.
{% hint style="info" %}
For predictable results, use it along with **dm:sort** tag.
{% endhint %}
In the below example, for a reference, we are going to use **VMware vROps** as an extension to query the data and ingest it into **dm:head** tag to list top 'n' rows.
Enter the below command to select **VMware vROps Datastore Summary tag (\*vrops:datastore\_summary**). (In this example, vrops name is used as a label to identify VMware vROps extension and it's tags. The label is defined while adding the extension in cfxdx configuration file or through UI)
```
tag *vrops:datastore_summary
```
**Example 1:** List top 10 rows
Get the data from VMware vROps Datastore Summary tag (**\*vrops:datastore\_summary**) and list top 10 rows.
```
data --> dm:head
```
**Example 2:** List top 20 rows
Get the data from VMware vROps Datastore Summary tag (**\*vrops:datastore\_summary**) and list top 20 rows.
```
data --> dm:head n = 20
```
**Example 3:** Sort the datastore by name (in ascending order) and List top 20 rows
Get the data from VMware vROps Datastore Summary tag (**\*vrops:datastore\_summary**), sort the data by datastore column 'name' and list top 20 rows.
```
data --> dm:sort columns = 'name' --> dm:head n = 20
```
# cfxdm - dm:tail
Get last 'n' rows
**dm:tail:** This cfxdm tag allows the user to fetch last 'n' rows from the queried data.
**dm:tail** syntax**:**
* **n (optional)**. Specify number of last rows that need to be listed. When this argument is not specified, by default it retrieves last 10 rows.
{% hint style="info" %}
For predictable results, use it along with **dm:sort** tag.
{% endhint %}
In the below example, for a reference, we are going to use **VMware vROps** as an extension to query the data and ingest it into **dm:tail** tag to list last 'n' rows.
Enter the below command to select **VMware vROps Datastore Summary tag (\*vrops:datastore\_summary**). (In this example, vrops name is used as a label to identify VMware vROps extension and it's tags. The label is defined while adding the extension in cfxdx configuration file or through UI)
```
tag *vrops:datastore_summary
```
**Example 1:** List last 10 rows
Get the data from VMware vROps Datastore Summary tag (**\*vrops:datastore\_summary**) and list last 10 rows.
```
data --> dm:tail
```
**Example 2:** Sort the data in ascending order and List last 20 rows.
Get the data from VMware vROps Datastore Summary tag (**\*vrops:datastore\_summary**), sort the data by datastore name and list last 20 rows.
```
data --> dm:sort columns = 'name' --> dm:tail n = 20
```
# cfxdm - dm:dedup
Remove duplicates from the data
**dm:dedup:** This cfxdm tag allows the user to remove the duplicate values from the queried data for a selected column or columns.
It can be used to find the unique values from a selected column or unique values from a more than one selected columns by looking at them as a combined value.
**dm:dedup** syntax**:**
* **columns (**optional**)**. Specify a column or columns (comma separated) on which de-duplication of the data to be applied.
In the below example, for a reference, we are going to use **VMware vROps** as an extension to query the data and ingest it into **dm:dedup** to remove any duplicate values from a selected column or columns.
Enter the below command to select **VMware vROps Datastore Summary tag (\*vrops:datastore\_summary**). (In this example, vrops name is used as a label to identify VMware vROps extension and it's tags. The label is defined while adding the extension in cfxdx configuration file or through UI)
```
tag *vrops:datastore_summary
```
**Example 1:** Remove duplicate datastore names from a selected column
Get the data from VMware vROps Datastore Summary tag (**\*vrops:datastore\_summary**) and remove duplicate datastore names from a selected column '**name**'
```
data --> dm:dedup columns = 'name'
```
**Example 2:** Find the unique values in the combination of two selected columns (**name** & **parent\_datacenter**)
When more than one column is selected, **dm:dedup** looks at the values of the selected columns as a single data string and find unique values.
Get the data from VMware vROps Datastore Summary tag (**\*vrops:datastore\_summary**) and find the unique values in the combination of two selected columns (**name** & **parent\_datacenter**)
```
data --> dm:dedup columns = 'name,parent_datacenter'
```
# cfxdm - dm:selectcolumns
Include or Exclude columns
**dm:selectcolumns:** This cfxdm tag allows the user to include or exclude columns using a regular expression format after retrieving the data from an extension's tag.
It is very useful when the user is dealing with many columns from an extension's tag and it simplifies columns select using either include or exclude or both options together
**dm:selectcolumns** syntax**:**
* **include (**optional**):** Specify a column or columns in regular expression format to include matched columns.
* **exclude (**optional**):** Specify a column or columns in regular expression format to exclude matched columns.
In the below example, for reference, we are going to use **VMware vROps** as an extension to query the data and ingest it into **dm:selectcolumns** to select specific columns using include/exclude or both together.
Enter the below command to select **VMware vROps Alerts tag (@vrops:alerts**). (In this example, vrops name is used as a label to identify VMware vROps extension and its tags. The label is defined while adding the extension in cfxdx configuration file or through UI)
```
tag @vrops:alerts
```
**Example 1:** Select matched columns using the include option
Get the data from VMware vROps Alerts tag (**@vrops:alerts**) and select any column that starts with **alert** and **start (below are all available columns from vrops:alerts tab)**
* **alertDefinitionId**
* **alertDefinitionName**
* **alertId**
* **alertImpact**
* **alertLevel**
* cancelTimeUTC
* controlState
* links
* resourceId
* **startTimeUTC**
* status
* subType
* suspendUntilTimeUTC
* type
* updateTimeUTC
```
data --> @dm:selectcolumns include = 'alert.*|start.*'
```
**Example 2:** Select matched columns using the include and exclude option
Get the data from VMware vROps Alerts tag (**@vrops:alerts**) and select any column that starts with **alertDefinitionName** or contains **Time,** but excludes **alertImpact** and **suspendUntilTimeUTC** columns using regular expression.
```
data --> @dm:selectcolumns include = 'alert.*.Name|.*Time.*' & exclude = 'suspend.*|.*Impact'
```
# cfxdm - dm:mergecolumns
Data merge from multiple columns to a single column
**dm:mergecolumns:** This cfxdm tag allows the user to select multiple columns using include or exclude columns options using a regular expression format and merge them into a single target column.
**dm:mergecolumns** syntax**:**
* **include (**Mandatory**):** Specify a column or columns in regular expression format to include matched columns.
* **exclude (**Optional**):** Specify a column or columns in regular expression format to exclude matched columns.
* **to** (Mandatory): Target column for merged data from selected one or more columns.
{% hint style="info" %}
**Note:** After merging multiple columns into a single column, it will remove source columns from the final output.
{% endhint %}
In the below example, for a reference, we are going to use **Elasticsearch** as an extension to query **Netflow data** and ingest it into **dm:mergecolumns** to select specific columns using include/exclude or both together and merge them into a single target column.
Enter the below command to select **Netflow tag (#es:netflow**). (In this example, **es** name is used as a label to identify Elasticsearch extension and it's tags that are pointing to Netflow data index. The label is defined while adding the extension in cfxdx configuration file or through UI)
```
tag #es:netflow
```
{% hint style="info" %}
Netflow tag includes many columns, in this exercise, we are going to use only few selective columns. For include / exclude regular expressions examples, please refer [**dm:selectcolumns**](/rda/rda-userguide/rda-data-management-cfxdm/cfxdm-dm-selectcolumns) documentation.
{% endhint %}
**Example 1:** Select three columns using include option from Netflow tag and merge them into a single column.
Get the TCP protocol data from Elasticsearch Netflow tag (**#es:netflow**) for last 1 hour and select the below three columns and merge them (values) together into a single target column.
**Source Columns:**
* flow\.client\_addr
* flow\.server\_addr
* flow\.service\_port
**Target Column:**
* flow\.unique**\_**id
Get the data with above three source columns for a quick data review.
```
data `flow.ip_protocol` contains 'TCP' and `@timestamp` after -1 hour GET `flow.client_addr`, `flow.server_addr`, `flow.service_port`
```
Extend the query by ingesting the above queried data selecting three columns into **dm:mergecolumns** tag, include all three columns and merge them into a single column as explained above.
```
data `flow.ip_protocol` contains 'TCP' and `@timestamp` after -1 hour GET `flow.client_addr`, `flow.server_addr`, `flow.service_port` --> dm:mergecolumns include = 'flow.client_addr|flow.server_addr|flow.service_port' & to = 'flow.unique_id'
```
# cfxdm - dm:describe
Show column statistics
**dm:describe:** This cfxdm tag allows the user to select one or more columns and provides below summarized statistics on the values within them.
* **count:** Total count of the values within a column
* **unique:** Total unique count of the values within a column
* **top:** The top value out of all unique values which is repeated compared to others
* **freq:** The number of times the top unique is repeated
* **mean:** The mean out of all of the values (applies to rows that only has numeric values)
* **std:** The standard deviation out of all of the values (applies to rows that only has numeric values)
* **min:** The minimum value out of all of the values (applies to rows that only has numeric values)
* **25%:** 25% percentile out of all of the values (applies to rows that only has numeric values)
* **50%:** 50% percentile out of all of the values (applies to rows that only has numeric values)
* **75%:** 75% percentile out of all of the values (applies to rows that only has numeric values)
* **max:** The maximum value out of all of the values (applies to rows that only has numeric values)
**dm:describe** syntax**:**
* **columns (**optional**)**. Specify a column or columns (comma separated) on which '**describe**' to be applied.
In the below example, for a reference, we are going to use **VMware vROps** as an extension to query the data and ingest it into **dm:selectcolumns** to select specific columns using include/exclude or both together.
Enter the below command to select **VMware vROps Alerts tag (@vrops:alerts**). (In this example, vrops name is used as a label to identify VMware vROps extension and it's tags. The label is defined while adding the extension in cfxdx configuration file or through UI)
```
tag @vrops:alerts
```
**Example 1:** Show describe statistics on all of the columns from the selected tag.
Get the data from VMware vROps Alerts tag (**@vrops:alerts**) and ingest the data into the **dm:describe** tag to see the statistics. (above mentioned in this document). Below are the columns from the **vrops:alerts** extension tag**.**
* alertDefinitionId
* alertDefinitionName
* alertId
* alertImpact
* alertLevel
* cancelTimeUTC
* controlState
* links
* resourceId
* startTimeUTC
* status
* subType
* suspendUntilTimeUTC
* type
* updateTimeUTC
```
data --> dm:describe
```
**Example 2:** Show describe statistics on selective columns from the selected tag.
Get the data from VMware vROps Alerts tag (**@vrops:alerts**) and ingest the data into the **dm:describe** tag to see the statistics on the below selected columns. (above mentioned in this document)
* alertDefinitionName
* startTimeUTC
```
data --> dm:describe columns = 'alertDefinitionName,startTimeUTC'
```
# cfxdm - dm:hist
Histogram on timeseries data
**dm:hist:** This cfxdm tag allows the user to generate an histogram out of timeseries data from a selected extension's tag.
**dm:hist** syntax**:**
* **timestamp (mandatory):** select timestamp column from the selected extension's tag.
* **interval (mandatory):** select the interval in days or hours or minutes or seconds. (Ex: 1d (day), 4h (hour), 15min (minutes), 30s (seconds)
In the below example, for a reference, we are going to use **Elasticsearch** as an extension to query the timeseries data index (or indexes) and ingest it into **dm:hist** to generate an histogram.
Enter the below command to select **Elasticsearch's index tag (#es124111:winlog-events**). (In this example, **es124111** name is used as a label to identify Elasticsearch extension, and **winlog-events** representing Windows log events indexes with Elasticsearch extension as a tag. The label is defined while adding the extension in cfxdx configuration file or through UI or CLI)
```
tag #es124111:winlog-events
```
**Example 1:** Select log events as timeseries data and generate an histogram
Get timeseries data out of Windows log events from Elasticsearch tag for last 10 hours with an interval of 1 hour and generate an histogram.
**Timestamp Column:**
* **@timestamp**
```
data `@timestamp` after -10 hour --> dm:hist timestamp = '@timestamp' & interval = '1h'
```
# cfxdm - dm:bin
Grouping the data in different bins
**dm:bin:** This cfxdm tag allows the user to group the numerical data into different bins,
**What is data binning ?** : Binning is a way to group numbers that are continuous values into a smaller number groups which are called as "bins". For example, if you have data about a group of people, you might want to arrange their ages into a smaller number of age groups. (5-15 years, 20-35 years, 45-65 years etc..)
**dm:bin** syntax**:**
* **column** (mandatory): Column name which has numerical values
* **bins** (mandatory): comma separated numerical range values
In the below example, for a reference, we are going to use **VMware vCenter** as an extension to query the data from VMs tag and their disk size.
Enter the below command to select **VMware vCenter VMs tag (\*vcenter:vms**). (In this example, **vcenter** name is used as a label to identify VMware vCenter extension and it's tags. The label is defined while adding the extension in cfxdx configuration file or through UI)
```
tag *vcenter:vms
```
**Example-1:**
Get VM's list and limit the ouput to VM's Name, Disk Size in KB and VM's BIOS UUID and save it as named dataset. (vm-disk-size)
```
data * get name as 'vm_name',disk_capacity_KB as 'disk_capacity_kb',id as 'bios_uuid' --> dm:save name = 'vm-disk-size'
```
Recall the named dataset (vm-disk-size) and convert the disk size to MB. (use [**dm:recall**](/rda/rda-userguide/rda-data-management-cfxdm/cfxdx-dm-recall) tag for the below query)
```
data name = 'vm-disk-size' --> dm:map attr = 'disk_capacity_kb' & func = 'evaluate' & expr = "disk_capacity_kb / 1024"
```
Create the '**bins**' for VM's disk size which fall between 40000,75000,100000,150000,200000 MB in size. (use [**dm:recall**](/rda/rda-userguide/rda-data-management-cfxdm/cfxdx-dm-recall) tag for the below query)
```
data name = 'vm-disk-size' --> dm:map attr = 'disk_capacity_kb' & func = 'evaluate' & expr = "disk_capacity_kb / 1024" --> dm:bin column = 'disk_capacity_kb' & bins = '40000,75000,100000,150000,200000'
```
# cfxdm - dm:fixcolumns
Remove special character from Column names
**dm:fixcolumns:** This cfxdm tag allows the user to remove the special characters like @,. (dot) etc from a column name. If there is a special character in between a column name (ex: First.Last), it replaces it with (underscore) (Ex: First\_Last)
**dm:fixcolumns** syntax**:** It doesn't require any arguments. Just ingest the data into this tag using a pipe (-->)
In the below example, for a reference, we are going to use **Netflow data that is ingested into Elasticsearch** as an extension to query the data and ingest it into **dm:mergecolumns** to select specific columns using include/exclude or both together and merge them into a single target column.
Enter the below command to select **Netflow tag (#es:netflow**). (In this example, **es** name is used as a label to identify Elasticsearch extension and it's tags that are pointing to Netflow data index. The label is defined while adding the extension in cfxdx configuration file or through UI)
```
tag #es:netflow
```
**Example 1:** Select three columns using the GET option from the Netflow tag.
Get the TCP protocol data from Elasticsearch Netflow tag (**#es:netflow**) for the last 1 hour and select the below three columns and ingest them into dm:fixcolumns tag to rename the names of the columns by replacing the special character (. (dot) with \_ (underscore).
**Source Columns:**
* flow\.client.addr
* flow\.server.addr
* flow\.service.port
**Output Columns:** (after replacing the special character ". (dot)" with "\_ (underscore)"
* flow\_client\_addr
* flow\_server\_addr
* flow\_service\_port
```
data * get `flow.client_addr`,`flow.server_addr`,`flow.service_port` --> dm:fixcolumns
```
# cfxdm - dm:save
Save data into a named dataset
**dm:save:** This cfxdm tag allows the user to save the retrieved data from an extension's tag as a named dataset for later consumption.
**dm:save** syntax**:** Below arguments are supported.
* **name** (mandatory): Name (unique) of the dataset which is to be saved.
* **publish** (optional): Name of the tag to ingest or publish into cfxDimensions platform. It can be used only with cfxDimensions platform configuration.
{% hint style="info" %}
This extension tag is typically used along with [**dm:recall** ](/rda/rda-userguide/rda-data-management-cfxdm/cfxdx-dm-recall)tag.
{% endhint %}
{% hint style="info" %}
To list the named datasets, use [**dm:savedlist**](/rda/rda-userguide/rda-data-management-cfxdm/cfxdm-dm-save) tag.
{% endhint %}
In the below example, for a reference, we are going to use **VMware vROps** as an extension to query the data and ingest it into **dm:selectcolumns** to select specific columns using include/exclude or both together.
Enter the below command to select **VMware vROps Alerts tag (@vrops:alerts**). (In this example, vrops name is used as a label to identify VMware vROps extension and it's tags. The label is defined while adding the extension in cfxdx configuration file or through UI)
```
tag @vrops:alerts
```
**Example 1:**
Get VMware vROps alert data and save it as a named dataset.
```
data --> dm:save name = 'vrops-alerts'
```
Use **dm:savedlist** tag to list saved datasets
```
tag dm:savedlist
```
```
data
```
# cfxdm - dm:savedlist
List named datasets
**dm:savedlist:** This cfxdm tag allows the user to list the saved named datasets.
**dm:savedlist** syntax**:** It doesn't require any arguments
{% hint style="info" %}
This tag is typically used along with [**dm:save**](/rda/rda-userguide/rda-data-management-cfxdm/cfxdm-dm-savedlist) tag.
{% endhint %}
Use **dm:savedlist** tag to list saved named datasets. In the below example, below are some of the named datasets which were saved using **dm:save** tag before.
* netflow-data
* vrops-alerts
```
tag dm:savedlist
```
```
data
```
# cfxdx - dm:recall
Recall or retrieve the data from saved named dataset
**dm:recall:** This cfxdm tag allows the user to recall or retrieve the data from a selected saved named dataset.
**dm:recall** syntax**:**
* **name** (mandatory)**:** Name of the dataset to recall - **name = '\'**
* **cache** (optional)**:** Cache the result for future recalls. 'yes' or 'no' - **cache = '\' (**default is 'no**').**
* **cache\_refresh\_seconds** (optional)**: cache\_refresh\_seconds = '\' (**default 120 seconds**).** Refresh the cache (if new update available) after specified seconds
* **return\_empty** (optional): Return an empty dataframe if an error occurs loading the dataset - **return\_empty = '\' (**default is 'no**').**
* **empty\_df\_columns** (optional): Comma seperated list of columns for empty dataframe.
Use [**dm:savedlist**](/rda/rda-userguide/rda-data-management-cfxdm/cfxdm-dm-savedlist) tag to list saved datasets
```
tag @dm:savedlist
```
```
data
```
Use **dm:recall** tag to retrieve the data from named dataset which was saved before using [**dm:save**](/rda/rda-userguide/rda-data-management-cfxdm/cfxdm-dm-save) tag.
```
tag @dm:recall
```
```
data name = 'vrops-alerts'
```
# cfxdm - dm:concat
Merge or append two or more named datasets
**dm:concat:** This cfxdm tag allows the user to merge two or more named datasets.
**dm:concat** syntax**:**
* **names** (mandatory)**:** List of two or more named datasets, supports regex
{% hint style="info" %}
Please refer [**dm:save**](/rda/rda-userguide/rda-data-management-cfxdm/cfxdm-dm-save) and [**dm:savedlist**](/rda/rda-userguide/rda-data-management-cfxdm/cfxdm-dm-savedlist) tags on how to create and list named datasets.
{% endhint %}
Use [**dm:savedlist**](/rda/rda-userguide/rda-data-management-cfxdm/cfxdm-dm-savedlist) tag to list saved datasets
```
tag dm:savedlist
```
```
data
```
**Example 1:**
From the above named datasets, merge **prtg-alerts** & **vrops-alerts** into one new named dataset using dm:concat tag
```
tag dm:concat
```
```
data names = "prtg-alerts|vrops-alerts" --> dm:save name = 'consolidated-alerts'
```
OR
```
data names = ".*.alerts|.*.alerts" --> dm:save name = 'consolidated-alerts'
```
Change the tag to **dm:savedlist** to view the newly created 'consolidated-alerts' named dataset.
```
tag dm:savedlist
```
```
data
```
# cfxdm - dm:groupby
Group rows by selected columns
**dm:groupby:** This cfxdm tag allows the user to group the data (by rows) based on selected columns using aggregate functions.
**dm:groupby** syntax**:**
* **columns** (mandatory)**:** Select one or more columns for grouping the data.
* **agg** (optional)**:**
* **count:** It is applied by default when 'agg' is not specified. Supported on any value types (numeric or non-numeric values)
* **min:** Supported on numeric values only
* **max:** Supported on numeric values only
* **sum:** Supported on numeric values only
In the below example, for a reference, we are going to use **VMware vCenter** as an extension to query the data from VMs tag and their disk size.
Enter the below command to select **VMware vCenter VMs tag (\*vcenter:vms**). (In this example, **vcenter** name is used as a label to identify VMware vCenter extension and it's tags. The label is defined while adding the extension in cfxdx configuration file or through UI)
```
tag *vcenter:vms
```
**Example-1:**
Get the VM's list from the above tag and select the below columns (as shown)
* vm\_name
* disk\_capacity\_kb
* bios\_uuid
```
data * get name as 'vm_name',disk_capacity_KB as 'disk_capacity_kb',id as 'bios_uuid'
```
Pipe the data to **dm:groupby** tag to view the data grouped by VM Name.
{% hint style="info" %}
Since '**agg**' option is not specified within the **dm:groupby** tag, it applies agg function '**count**' by default.
{% endhint %}
```
data * get name as 'vm_name',disk_capacity_KB as 'disk_capacity_kb',id as 'bios_uuid' --> dm:groupby columns = 'vm_name'
```
**Example-2:**
Continuing with above data, lets use agg function '**sum**' to calculate total VM's disk size.
{% hint style="info" %}
When agg function '**sum**' is used, it automatically applies on all eligible columns (numerical values).
{% endhint %}
```
data * get name as 'vm_name',disk_capacity_KB as 'disk_capacity_kb',id as 'bios_uuid' --> dm:groupby columns = 'vm_name' & agg = 'sum'
```
---
[Next Page](/llms-full.txt/1)