CloudFabrix Documentation
Asset IntelligenceOps IntelligenceObservabilityRobotic Data
  • Getting Started
  • CloudFabrix Overview
  • AIOps Platform
  • AIOps Solutions
  • CloudFabrix RDA
    • RDA - Overview
      • RDA - Terminology and Artifacts
    • RDA - Installation
      • Linux OS
      • Windows OS
      • Mac OS
      • RDA Client
      • Worker Nodes
      • Event Gateway
      • Edge Collector
      • Log Shippers
        • Filebeat
        • Fluentd
        • Rsyslog
        • Syslog (udp)
        • Splunk forwarder (Windows and Linux)
        • Winlogbeat (Windows)
      • RDA Log Archives
    • RDA - Administration
      • RDA - Configuration
    • RDA - User Guide
      • RDA - AIOps Studio
        • AIOps Studio - Solution Packages
        • AIOps Studio - Pipelines
        • AIOps Studio - Explore
        • AIOps Studio - Administration
        • RDA CLI in UI
        • AIOps Studio - Examples
          • File Operations
          • Loop Operations
          • Data Management Operations - cfxdm
          • Data mapping - cfxdm - dm:eval
          • Filters - cfxdm - dm:filter
          • Data mapping - cfxdm - dm:map
          • Metadata - cfxdm-dm:metadata
          • Data mapping - cfxdm - dm:functions
            • Any_non_null
            • Concat
            • Datetime
            • Date and Timestamp
            • Evaluate
            • Fixed
            • Highest
            • Join
            • jsonDecode
            • Lower
            • Lowest
            • Match
            • Minutes_Between
            • Replace
            • Seconds_Between
            • Slice
            • Split
            • Strip
            • To_Numeric
            • Ts_To_Datetimestr
            • Upper
            • When_Null
          • Data Mapping cfxdm - dm:sort
          • Data Mapping cfxdm - dm:head
          • Data Mapping cfxdm - dm: tail
          • Data Mapping cfxdm - dm: dedup
          • Data Mapping cfxdm - dm:selectcolumns
          • Data Mapping cfxdm - dm:fixcolumns
          • Data Mapping cfxdm - dm:mergecolumns
          • Data Mapping cfxdm - dm:describe
          • Data Mapping cfxdm - dm:save
          • Data Mapping cfxdm - dm:savedlist
          • Data Mapping cfxdm - dm:recall
          • Data Mapping cfxdm - dm:concat
          • Data Mapping cfxdm - dm:groupby
          • Data Mapping cfxdm - dm:to_type
          • Data Mapping cfxdm - dm:enrich
          • Data Mapping cfxdm - dm:dns_ip_to_name
          • Data Mapping cfxdm - dm:dns_name_to_ip
        • AIOps Studio - Datasource Examples
          • Elasticsearch (v1)
      • RDA - Data Management (cfxdm)
        • cfxdm - dm:filter
        • cfxdm - dm:map
        • cfxdm - dm:functions
        • cfxdm - dm:sort
        • cfxdm - dm:head
        • cfxdm - dm:tail
        • cfxdm - dm:dedup
        • cfxdm - dm:selectcolumns
        • cfxdm - dm:mergecolumns
        • cfxdm - dm:describe
        • cfxdm - dm:hist
        • cfxdm - dm:bin
        • cfxdm - dm:fixcolumns
        • cfxdm - dm:save
        • cfxdm - dm:savedlist
        • cfxdx - dm:recall
        • cfxdm - dm:concat
        • cfxdm - dm:groupby
        • cfxdm - dm:enrich
        • cfxdm - dm:to_type
        • cfxdm - dm:dns_ip_to_name
        • cfxdm - dm:dns_name_to_ip
        • cfxdm - files:loadfile
      • RDA Terminal
        • Examples using Terminal / Commandline
    • RDA - Bot Documentation
    • RDA - Datasource Integrations
      • AppDynamics
      • Dynatrace
      • Dell EMC Unity
      • Elasticsearch
      • Infoblox NetMRI
      • Kubernetes Cluster
      • Linux OS
      • Microsoft Windows Server OS
      • Nagios XI
      • NetApp Clustered ONTAP
      • PRTG Network Monitor
      • VMware vCenter
      • VMware vRealize Operations
    • RDA - Python API
      • Class CaaSDataset
      • Class CaaSClient
      • Python API Example
    • RDA - FAQ
      • Download and Installation
      • Troubleshooting
  • CFXQL - CFX Query Language
    • CFXQL User Interface
  • Operations Intelligence Analytics (OIA)
    • Solution Overview
      • Navigating cfxOIA
      • Any title
    • Administration Guide
      • Active Directory Integration
      • Add Customer
      • First Steps
      • OIA Users
      • Add New Users and Assign Roles
      • Add Environment
      • Add Gateway
      • Add DataSource
      • Add Project
      • Project Configuration
      • Stacks
      • Teams
    • User Guide
      • Incidents
        • Incident
        • Stack
      • Alerts
      • Outcomes
  • INTEGRATIONS
    • Zabbix
    • AppDynamics
    • Dynatrace
    • NetApp C Mode
    • NetApp 7 Mode
    • Splunk Enterprise
    • VMware vCenter
  • OBSERVABILITY
  • Observability - IT Infrastructure Monitoring (cfxPulse)
    • Getting Started
    • Solution Overview
    • Installing cfxPulse Collector
    • Administrator Quick Start Guide
      • Prerequisites
      • Accessing cfxPulse
      • Configuration of cfxPulse
      • Setting Up Monitoring
      • Monitoring Using Prometheus Agents
      • How to add Prometheus Agent Details
      • Discovery of Devices
      • Monitoring Dashboard
    • End User Quick Start Guide
      • Portal Navigation
      • Monitoring Analysis
      • Monitoring
      • Alerts and Incidents
      • Reports
      • NOC/Ops
      • Configuration Backups
      • Interacting With Tabular Reports
      • Creating Custom Dashboards
  • Observability - Log Monitoring & Analytics (CFX LogAnalytics or CLA)
    • Getting Started
    • Solution Overview
    • Solution Key Components
    • Log Forwarding
      • Install and Configure Logstash
      • Sending Logs to Logstash Forwarder
    • Log Collection
      • Collecting Logs from Linux
    • Log Transformation & Enrichment
    • Logstash Installation
      • How to install Java / Logstash on client side
  • Asset Intelligence Analytics (AIA) Solution
    • Getting Started
    • Solution Overview
    • AIA Roles
      • Platform Admin
        • Managed Service Provider (MSP)
        • Authentication Server
        • Set Up Services
        • Organizations
        • Users
      • Organization Admin
        • My Organizations
      • Organization Executive
      • Organization User
    • AIA Tasks, Functions
      • Home Page Navigation
      • Filters
      • Settings Menu
      • Notifications
      • Authentical Server
      • How to Add, Edit, Delete MSP
      • Actions
        • Services
        • Files
        • Dictionaries
        • Discovery Jobs
        • Snapshots
        • Clambda Jobs
        • State Operations
        • Replacement Rate
      • Details
        • Overall
        • POR Insights
        • HW Assets
        • SW Assets
        • Contracts
        • App Dependency
        • Asset List
    • AIA API
    • Enterprise Discovery
      • cfxEdgeCollector
        • Deployment of cfxEdgeCollector
        • Configuration of cfxEdgeCollector
        • cfxEdgeCollector Command Line Options
        • cfxEdgeCollector Help Command
        • Working With cfxEdgeCollector
        • cfxEdgeCollector Auto Export
    • Asset Intelligence & Analytics (AIA) (Delete)
  • CloudFabrix SaaS
    • Signup
    • Navigation
    • User Roles
  • Support
    • Contact Support
Powered by GitBook
On this page
  1. CloudFabrix RDA
  2. RDA - User Guide
  3. RDA - Data Management (cfxdm)

cfxdm - dm:functions

Data manipulation and transformation

dm:functions: This cfxdm tag provides very comprehensive data manipulation & transformation functions and below are the details about them and their usage.

  • any_non_null: Returns any non-null value from a list of input values, @param value is optional, if not specified, returns None when none of the listed values meet the criteria. Input must be a list (else treated as a single item list).

  • concat: Adds prefix and suffix to the specified string, @param prefix type is string (optional). @param suffix type is a string (optional). Input must be a string. If the input is null, it is treated as ' '

  • datetime: Parses input string and converts into an epoch milliseconds format number. Input must be a string. @param tzmap: type dict (optional). Dictionary of timezone mappings from custom/local to standard timezones. @param expr: type string (optional). A custom timestamp format with UTC/local timezone.

  • evaluate: Given an expression evaluates the expression string. If performed on the dataframe row, it evaluates by passing the row as a dictionary. If performed on a single value, it expects an additional argument 'key' to be used in the expression. @param expr: The expression to evaluate. @param key: An optional 'key' if evaluated on a single value instead of a dictionary.

  • fixed: Returns a fixed value specified by the 'value' parameter. @param value Type can be string or number. Input can be of any type.

  • formDecode: Decodes input string to remove any URL encoded values. Requires no parameters. Input must be a string.

  • highest: Returns highest non-null value from the list of integer values. @param default (optional), Type int. If provided none of the input values are non-null, returns default Input must be a number or list of numbers.

  • hours_between: Number of hours between two datetime strings. If only one specified, compare diff between now and that timestamp.

  • join: Joins input list using an optional separator. @param sep (optional), default value is ' ' . Input is expected to be a list. If the input is not a list, it returns the value without joining.

  • jsonDecode: Decodes input string into JSON object. Requires no parameters. Input must be a string.

  • lower: Converts to lowercase text, requires no parameters. Input must be a string.

  • lowest: Returns lowest non-null value from the list of int values. @param default (optional), Type int. If provided none of the input values are non-null, returns default, Input must be a number or list of numbers.

  • map_values: Maps input value using the specified name value dictionary. If no values match and ""*"" key is provided, it returns the ""*"" key's values, or else the original value will be maintained. Input must be a string.

  • match: Matches a regular expression and extracts a specific value (if matched). @param expr Type string. Regular expression @param flags List of optional flags (A I M L S X) Input must be a string.

  • minutes_between: Number of minutes between two date-time strings. If only one is specified, it compares the difference between it and the current timestamp.

  • replace: Replaces old value with new value in the input string @param oldvalue, Type string. @param new value, Type string. Input must be a string.

  • seconds_between: Number of seconds between two datetime strings. If only one is specified, it compares the difference between it and the current timestamp.

  • slice: Slices a string or an array using specified indices. @param from-index Type int. Default value 0 @param to-index Type int. The default value is None. Input can be a string or a list. If neither, it converts input to a string.

  • split: Splits the input using specified 'sep' separator. @param sep Type string. Optional. Default any whitespace characters. Input must be a string.

  • strip: Strips white spaces from both sides of a string, Requires no parameters. Input must be a string.

  • timediff:

  • to_numeric: Convert input value into numeric

  • ts_to_datetimestr: Processes input number with specified 'unit' (s,ms,ns,excel_date) and converts the value to datetime string specified by 'format', default is ISO format. Input must be a float or int. @param 'unit' (Type string), must be s,ms,ns,excel_date, default is 'ms' @param 'format' (Type string), default is None (ISO format)

  • upper: Converts to uppercase text Requires no parameters. Input must be a string.

  • valueRef: Extracts a specific item from the input dictionary object. @param path A dot '.' delineated path to the element within the dictionary, Input must be dictionary object.

  • when_null: If the specified value is null, it uses the value as per 'value' param @param value Type can be string or number. Input can be of any type.

Examples:

Convert Milliseconds to Human readable date & timestamp

Functions:

  • datetime

  • ts_to_datetimestr

In the below syntax example, 'Alert_Time' is a column name that has a timestamp value in milliseconds format.

dm:map attr = 'Alert_Time' & func = 'ts_to_datetimestr' & unit = 'ms'

From: (Time in Milliseconds)

To: (Time in Date & Time format)

In the above example, vrops-alerts is a named dataset that was created out of the VMware vROps extension's alerts tag.

Convert Human readable Date & Timestamp to Milliseconds

dm:map attr = 'Alert_Time' & func = 'datetime'

From: (Time in Date & Time format)

To: (Time in Milliseconds)

In the above example, prtg-alerts is a named dataset that was created out of the PRTG Monitor extension's alerts tag.

Join values from two to more columns into one.

Function:

  • join

In the below syntax example, 'resource_kind & adapter_kind' are columns and their values are joined together in a new column called 'resource_and_vendor' using the function 'join' (optional separator '-' is used in this example, when it is not specified, the default separator is space.

dm:map from = 'resource_kind,adapter_kind' & to = 'resource_and_vendor' & func = 'join' & sep = '-'

Source Columns & their values:

Destination Column (after join)

Arithmetic Operations

Function:

  • Evaluate

To perform arithmetic operations on columns, make sure the column's data type is set to 'Integer' (Numeric) or 'Float'. Additionally, column values should not contain NULL or Empty values.

Multiplication: In the below example, vm_cpu_sockets & vm_cpu_cores are columns which has numeric values, using 'evaluate' function, multiplied 2 column values together and saving the result in new column call 'vm_cpu_total'.

dm:map to ='vm_cpu_total' & func = 'evaluate' & expr = 'vm_cpu_sockets * vm_cpu_cores'

Bytes to GB: In the below example, vm_disk_size_bytes is a column that has disk capacity in bytes, using the 'evaluate' function, create a new column 'vm_disk_size_gb' and convert the bytes into GB.

dm:map from = 'vm_disk_size_bytes' & to ='vm_disk_size_gb' --> dm:map attr = 'vm_disk_size_gb' & func = 'evaluate' & expr = 'vm_disk_size_gb / 1024 / 1024 / 1024'
Previouscfxdm - dm:mapNextcfxdm - dm:sort

Last updated 3 years ago