Elasticsearch (v1)
Elascticsearch integration with AIOps/RDA.
Last updated
Elascticsearch integration with AIOps/RDA.
Last updated
This section explains how to add Elasticsearch data source, ingest data into Elasticsearch and query the data using AIOps/RDA environment.
RDA's user interface is used to configure Elasticsearch data source.
Step 1: Accessing RDA UI
Login into RDA's user interface using a browser.
https://<rda-ip-address>:9998
Under 'Notebook', click on 'CFXDX Python 3' box
Step 2: Adding Elasticsearch data source instance to RDA/AIOps
In the 'Notebook' command box, type botadmin() and alt (or option) + Enter to open the data source administration menu.
Click on the 'Add' menu and under Type drop-down, select elasticsearch
Type: Datasource/Extension type. In this context, it is 'elasticsearch'
name: Datasource/Extension label which should be unique within the RDA
Hostname: Elasticsearch IP Address or FQDN/DNS name
Username: User account that was created with 'read-only' permissions
Password: User account's password
Click on 'Check Connectivity' to verify the network access and credentials validity from RDA to Elasticsearch instance. Once it is validated, click on the 'Add' button to add Elasticsearch as the data source
Step 3: Adding tag definition in RDA and associate with Elasticsearch index
Once the user completes Step 2 and checks/validates connectivity from RDA to elasticsearch, the user can now add/define a tag in RDA which maps to elasticsearch index that was created earlier.
In the 'Notebook' command box, type botadmin() and alt (or option) + Enter to open the data source administration menu.
Click on the 'Edit' menu and under Type drop-down, select 'es/elasticsearch' item that was created in step 2 (as shown in the below screenshot).
Note: In the above RDA tags definition, RDA keeps track of tag (rda-to-elasticsearch) to that of elasticsearch index (rda_to_elasticsearch_idx with unique id as idx).
The code snippet is captured in the below code block.
Note: Before performing step 3, make sure elasticsearch index (rda_to_elasticsearch_idx) has been created ahead in elasticsearch instance and verified using standard tools (e.g. curl or postman)
Step 4: Adding data using RDA and storing in Elasticsearch using the mapping that was created
Create a pipeline "rda_to_elasticsearch_example_1" and copy the below code into your pipeline and perform the rest of the steps in your environment.
##### This pipeline creates couple of user names and ids using RDA/AIOps Studio.
##### RDA uses the mapping that was created and stores the records into elasticsearch
--> @dm:empty
--> @dm:addrow idx = 1 & name = 'David' & lastname = 'Eiger' & email = 'deiger@hello.com'
--> @dm:addrow idx = 2 & name = 'Emma' & lastname = 'Edge' & email = 'eedge@hello.com'
--> @dm:addrow idx = 3 & name = 'John' & lastname = 'Seagul' & email = 'jseagul@hello.com'
--> @dm:addrow idx = 4 & name = 'Peter' & lastname = 'Samuel' & email = 'psamuel@hello.com'
--> @dm:addrow idx = 5 & name = 'Sean' & lastname = 'Taylor' & email = 'staylor@hello.com'
--> #es:rda-to-elasticsearch
Step 5: Verify the above-added pipeline using AIOps/RDA by selecting the 'Verify' button as shown in the below screenshot
Step 6: Execute the pipeline by selecting the 'Execute' button as shown in the below screenshot
Step 7: Verify the data stored in Elasticsearch using RDA (and or using curl command).
Method A -- Using Curl command
Step 1: Log in to the machine where Elasticsearch instance is running using putty or any other SSH tool
Step 2: Once you log in, run the following curl command to validate the data stored
The above curl command will return the data as pretty formatted JSON output as shown below.
Method B -- Using RDA pipeline
Step 1: Create a pipeline "verify_elasticsearch_to_rda_data_01" and copy the below code into your pipeline and perform the rest of the steps in your environment.
##### This pipeline verifies the data stored via RDA pipeline
#####
--> @c:new-block
--> #es:rda-to-elasticsearch
Step 2: Verify the above-created pipeline using RDA/AIOps studio as shown in the below screenshot
Step 3: Execute and verify the output of the data using RDA/AIOps studio as shown in the below screenshot
The above example walks through Elasticsearch integration with RDA using a simple inline dataset creation of users (name, last name, etc.). In addition, the datasets can come from files and/or other data sources like MySQL, etc. Users can explore other data sources using the above-explained steps.
Also, in the above example, a single Elasticsearch index 'idx' has been used to walk through the use case. Users can also extend or add additional indices to make a unique index based on the use case in context.
