Links

Elasticsearch (v1)

Elascticsearch integration with AIOps/RDA.

Introduction

This section explains how to add Elasticsearch data source, ingest data into Elasticsearch and query the data using AIOps/RDA environment.

Adding Elasticsearch as Datasource in 'RDA':

RDA's user interface is used to configure Elasticsearch data source.
Step 1: Accessing RDA UI
Login into RDA's user interface using a browser.
https://<rda-ip-address>:9998
Under 'Notebook', click on 'CFXDX Python 3' box
Step 2: Adding Elasticsearch data source instance to RDA/AIOps
In the 'Notebook' command box, type botadmin() and alt (or option) + Enter to open the data source administration menu.
Click on the 'Add' menu and under Type drop-down, select elasticsearch
Adding Elasticsearch data source to RDA/AIOps
  • Type: Datasource/Extension type. In this context, it is 'elasticsearch'
  • name: Datasource/Extension label which should be unique within the RDA
  • Hostname: Elasticsearch IP Address or FQDN/DNS name
  • Username: User account that was created with 'read-only' permissions
  • Password: User account's password
Click on 'Check Connectivity' to verify the network access and credentials validity from RDA to Elasticsearch instance. Once it is validated, click on the 'Add' button to add Elasticsearch as the data source
Step 3: Adding tag definition in RDA and associate with Elasticsearch index
Once the user completes Step 2 and checks/validates connectivity from RDA to elasticsearch, the user can now add/define a tag in RDA which maps to elasticsearch index that was created earlier.
In the 'Notebook' command box, type botadmin() and alt (or option) + Enter to open the data source administration menu.
Click on the 'Edit' menu and under Type drop-down, select 'es/elasticsearch' item that was created in step 2 (as shown in the below screenshot).
RDA tag 'rda-to-elasticsearch' to Elasticsearch index 'rda_to_elasticsearch_idx' mapping
Note: In the above RDA tags definition, RDA keeps track of tag (rda-to-elasticsearch) to that of elasticsearch index (rda_to_elasticsearch_idx with unique id as idx).
The code snippet is captured in the below code block.
- tag: rda-to-elasticsearch
index: rda_to_elasticsearch_idx
update:
index: rda_to_elasticsearch_idx
ids:
- idx
Note: Before performing step 3, make sure elasticsearch index (rda_to_elasticsearch_idx) has been created ahead in elasticsearch instance and verified using standard tools (e.g. curl or postman)
Step 4: Adding data using RDA and storing in Elasticsearch using the mapping that was created
Create a pipeline "rda_to_elasticsearch_example_1" and copy the below code into your pipeline and perform the rest of the steps in your environment. ##### This pipeline creates couple of user names and ids using RDA/AIOps Studio. ##### RDA uses the mapping that was created and stores the records into elasticsearch --> @dm:empty --> @dm:addrow idx = 1 & name = 'David' & lastname = 'Eiger' & email = '[email protected]' --> @dm:addrow idx = 2 & name = 'Emma' & lastname = 'Edge' & email = '[email protected]' --> @dm:addrow idx = 3 & name = 'John' & lastname = 'Seagul' & email = '[email protected]' --> @dm:addrow idx = 4 & name = 'Peter' & lastname = 'Samuel' & email = '[email protected]' --> @dm:addrow idx = 5 & name = 'Sean' & lastname = 'Taylor' & email = '[email protected]' --> #es:rda-to-elasticsearch
Pipeline added to injest data from RDA to Elasticsearch
Step 5: Verify the above-added pipeline using AIOps/RDA by selecting the 'Verify' button as shown in the below screenshot
Verify button will validate the syntax of pipeline
Step 6: Execute the pipeline by selecting the 'Execute' button as shown in the below screenshot
Successful execution of pipeline RDA to Elasticsearch storing data
Step 7: Verify the data stored in Elasticsearch using RDA (and or using curl command).
Method A -- Using Curl command
Step 1: Log in to the machine where Elasticsearch instance is running using putty or any other SSH tool
Step 2: Once you log in, run the following curl command to validate the data stored
curl -X GET 'http://localhost:9200/rda_to_elasticsearch_idx/_search'?pretty=true
The above curl command will return the data as pretty formatted JSON output as shown below.
Curl command returns all the records which are stored via execution of pipeline
Method B -- Using RDA pipeline
Step 1: Create a pipeline "verify_elasticsearch_to_rda_data_01" and copy the below code into your pipeline and perform the rest of the steps in your environment.
##### This pipeline verifies the data stored via RDA pipeline ##### --> @c:new-block --> #es:rda-to-elasticsearch
Step 2: Verify the above-created pipeline using RDA/AIOps studio as shown in the below screenshot
RDA queries data from Elasticsearch and outputs the stored records.
Step 3: Execute and verify the output of the data using RDA/AIOps studio as shown in the below screenshot
Execution of pipeline without any errors
RDA prints the data as shown in the above screenshot
The above example walks through Elasticsearch integration with RDA using a simple inline dataset creation of users (name, last name, etc.). In addition, the datasets can come from files and/or other data sources like MySQL, etc. Users can explore other data sources using the above-explained steps.
Also, in the above example, a single Elasticsearch index 'idx' has been used to walk through the use case. Users can also extend or add additional indices to make a unique index based on the use case in context.