Elasticsearch
Read, Update, Append data from or to Indices
Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.
CloudFabrix RDA provides out of the box integration for Elasticsearch through it's API interface. As part of the integration, it provides an ability to query the data from Elasticsearch indice(s), update or append the data into an Elasticsearch indice. Below Elastichsearch distributions are supported.
  • Elasticsearch Commercial & Open-source versions
  • Opendistro & Opensearch from AWS versions
Elasticsearch Permissions: Below permissions are required as a prerequisite.
  • read: To read, search/filter the data from indices
  • write: To create, update & append data into indices (optional)

Adding Elasticsearch as Datasource/Extension in 'RDA':

Elasticsearch or any other datasource/extension's configuration is configured in RDA's user interface. Login into RDA's user interface using a browser.
https://<rda-ip-address>:9998
Under 'Notebook', click on 'CFXDX Python 3' box
In the 'Notebook' command box, type botadmin() and alt (or option) + Enter to open datasource administration menu.
Click on 'Add' menu and under Type drop down, select elasticsearch_v2
Note: Elasticsearch extension typeelasticsearchis deprecated, please useelasticsearch_v2instead.
  • Type: Datasource/Extension type. In this context, it is 'elasticsearch_v2'
  • name: Datasource/Extension label which should be unique within the RDA
  • Hostname: Elasticsearch's IP Address or DNS name
  • URL Prefix: Use this option when Elasticsearch is behind a load balancer and it has additional path to the root (ex: /elasticsearch) - Optional
  • Username: Username that has read/write permissions to Elasticsearch indices (optional)
  • Password: User account's password (optional)
  • HTTP(s) Port: default is 9200, but can be changed to 443 or to other port
  • Protocol: API integration over HTTP/HTTPs protocol
  • Timeout(seconds): HTTP response timeout in seconds, default value is 30 seconds
Below are available data bots for Elasticsearch datasource.
Note: In the below example, elasticsearch datasource is added as 'esv2' and it is for a reference only.
To list Elasticsearch indices:
1
> bot *esv2:list-indices
2
3
*esv2:list-indices> data
Copied!
To list metadata of Elasticsearch indice:
1
> bot @esv2:index-meta
2
3
Capabilities: get-data
4
API Model for this tag:
5
+----+--------+-------------+--------------+-----------+---------------------------+
6
| | name | mandatory | value_type | default | help |
7
|----+--------+-------------+--------------+-----------+---------------------------|
8
| 0 | index | Y | constant | | Index to get metadata for |
9
+----+--------+-------------+--------------+-----------+---------------------------+
Copied!
1
@esv2:index-meta> data index = 'filebeat_log_fortigate*'
Copied!
To query and read the data from Elasticsearch indice:
Elasticsearch bot read-index acts as a data sink where it expects the below parameters as inputs
  • index - Indice name, wildcard like * is supported (mandatory)
  • limit (by default it is set to 1000 records, it remove the limit, it need to be set to '0' (optional)
Example-1: In the below example provides how to pass index param as input to read-index bot. It queries the data from the given index pattern called 'filebeat_log*' (Note: It fetches 1000 records as the param limit is not specified and it's default value is set to 1000)
1
> bot @dm:empty
2
3
@dm:empty> data --> @dm:addrow index='filebeat_log*' --> #esv2:read-index
Copied!
Example-2: In the below example it queries the data from the given index pattern called 'filebeat_log* for last 24 hours from a hostname called linux-fin-db01
1
> bot @dm:empty
2
3
@dm:empty> data --> @dm:addrow index='filebeat_log*' & limit = '0' \
4
--> #esv2:read-index `@timestamp` after -24h and `agent.name` = 'linux-fin-db01'
Copied!
To update or add a new record in Elasticsearch indice:
Elasticsearch bot update-index requires the below parameters as inputs. Additionally it acts as a data sink and the data that need to be added / updated to an existing index should come from a different bot such as @dm:recall or file-load or other datasource.
  • index - Indice name (mandatory).
  • ids - One or more field names (comma separated) to be used as document id. When the given document id matches to an existing record, it updates it with new data.
When a given Indice name doesn't exist on Elasticsearch, it creates a new one and updates the record. Indice name support appending it with date attributes like DD, MM, YYYY. Below is an example.
index-name-%d-%m-%Y creates index-name-29-09-2021 based on UTC timestamp on the system.
To append a new record to Elasticsearch indice:
Elasticsearch bot append-index requires below parameter as input. Additionally it acts as a data sink and the data that need to appended to an existing index should come from a different bot such as @dm:recall or file-load or other datasource.
  • index - Indice name (mandatory).
Last modified 1mo ago