Elasticsearch (v1)

Elascticsearch integration with AIOps/RDA.

Introduction

This section explains how to add Elasticsearch data source, ingest data into Elasticsearch and query the data using AIOps/RDA environment.

Adding Elasticsearch as Datasource in 'RDA':

RDA's user interface is used to configure Elasticsearch data source.

Step 1: Accessing RDA UI

Login into RDA's user interface using a browser.

https://<rda-ip-address>:9998

Under 'Notebook', click on 'CFXDX Python 3' box

Step 2: Adding Elasticsearch data source instance to RDA/AIOps

In the 'Notebook' command box, type botadmin() and alt (or option) + Enter to open the data source administration menu.

Click on the 'Add' menu and under Type drop-down, select elasticsearch

Adding Elasticsearch data source to RDA/AIOps

• Type: Datasource/Extension type. In this context, it is 'elasticsearch'

• name: Datasource/Extension label which should be unique within the RDA

• Hostname: Elasticsearch IP Address or FQDN/DNS name

• Username: User account that was created with 'read-only' permissions

• Password: User account's password

Click on 'Check Connectivity' to verify the network access and credentials validity from RDA to Elasticsearch instance. Once it is validated, click on the 'Add' button to add Elasticsearch as the data source

Step 3: Adding tag definition in RDA and associate with Elasticsearch index

Once the user completes Step 2 and checks/validates connectivity from RDA to elasticsearch, the user can now add/define a tag in RDA which maps to elasticsearch index that was created earlier.

In the 'Notebook' command box, type botadmin() and alt (or option) + Enter to open the data source administration menu.

Click on the 'Edit' menu and under Type drop-down, select 'es/elasticsearch' item that was created in step 2 (as shown in the below screenshot).

RDA tag 'rda-to-elasticsearch' to Elasticsearch index 'rda_to_elasticsearch_idx' mapping

Note: In the above RDA tags definition, RDA keeps track of tag (rda-to-elasticsearch) to that of elasticsearch index (rda_to_elasticsearch_idx with unique id as idx).

The code snippet is captured in the below code block.

- tag: rda-to-elasticsearch
  index: rda_to_elasticsearch_idx
  update:
    index: rda_to_elasticsearch_idx
    ids:
    - idx

Note: Before performing step 3, make sure elasticsearch index (rda_to_elasticsearch_idx) has been created ahead in elasticsearch instance and verified using standard tools (e.g. curl or postman)

Step 4: Adding data using RDA and storing in Elasticsearch using the mapping that was created

Create a pipeline "rda_to_elasticsearch_example_1" and copy the below code into your pipeline and perform the rest of the steps in your environment. ##### This pipeline creates couple of user names and ids using RDA/AIOps Studio. ##### RDA uses the mapping that was created and stores the records into elasticsearch --> @dm:empty --> @dm:addrow idx = 1 & name = 'David' & lastname = 'Eiger' & email = '[e-mail]' --> @dm:addrow idx = 2 & name = 'Emma' & lastname = 'Edge' & email = '[e-mail]' --> @dm:addrow idx = 3 & name = 'John' & lastname = 'Seagul' & email = '[e-mail]' --> @dm:addrow idx = 4 & name = 'Peter' & lastname = 'Samuel' & email = '[e-mail]' --> @dm:addrow idx = 5 & name = 'Sean' & lastname = 'Taylor' & email = '[e-mail]' --> #es:rda-to-elasticsearch

Pipeline added to injest data from RDA to Elasticsearch

Step 5: Verify the above-added pipeline using AIOps/RDA by selecting the 'Verify' button as shown in the below screenshot

Verify button will validate the syntax of pipeline

Step 6: Execute the pipeline by selecting the 'Execute' button as shown in the below screenshot

Successful execution of pipeline RDA to Elasticsearch storing data

Step 7: Verify the data stored in Elasticsearch using RDA (and or using curl command).

Method A -- Using Curl command

Step 1: Log in to the machine where Elasticsearch instance is running using putty or any other SSH tool

bash# ssh macaw@10.95.103.111 

Step 2: Once you log in, run the following curl command to validate the data stored

curl -X GET 'http://localhost:9200/rda_to_elasticsearch_idx/_search'?pretty=true

The above curl command will return the data as pretty formatted JSON output as shown below.

Curl command returns all the records which are stored via execution of pipeline

Method B -- Using RDA pipeline

Step 1: Create a pipeline "verify_elasticsearch_to_rda_data_01" and copy the below code into your pipeline and perform the rest of the steps in your environment.

##### This pipeline verifies the data stored via RDA pipeline ##### --> @c:new-block --> #es:rda-to-elasticsearch

Step 2: Verify the above-created pipeline using RDA/AIOps studio as shown in the below screenshot

RDA queries data from Elasticsearch and outputs the stored records.

Step 3: Execute and verify the output of the data using RDA/AIOps studio as shown in the below screenshot

Execution of pipeline without any errors

RDA prints the data as shown in the above screenshot

The above example walks through Elasticsearch integration with RDA using a simple inline dataset creation of users (name, last name, etc.). In addition, the datasets can come from files and/or other data sources like MySQL, etc. Users can explore other data sources using the above-explained steps.

Also, in the above example, a single Elasticsearch index 'idx' has been used to walk through the use case. Users can also extend or add additional indices to make a unique index based on the use case in context.