Splunk Enterprise
Splunk Enterprise as Data Source
Prerequisites:
Splunk Enterprise Version 8.0.0
Splunk Indexes (Please refer Splunk Documentation on how to create indexes)
a. Eg: Webserver host logs created with log index “web_error_logs” with field values hostname, severity, ipaddress etc...
b. DB Server host logs created with log index “db_error_logs” with field values hostname, severity, ipaddress etc...
Note-1: For field “severity”, you can extract from the log message (Eg: info, error, warn, debug etc)
Note-2: Fields “hostname” and “ipaddress” should be the HostName and IP address of the host on which Splunk agent is installed and collecting the OS or Application logs.
To add the above custom fields on Splunk Agent follow the below steps (Linux version).
3. Enable API port 8089 on Splunk Enterprise administration UI
Enabling API Port 8089
Step 1: Log into Your Splunk Enterprise Instance with admin credentials
Step 2: From Top Menu bar, click on 'Settings' tab.
Step 3: Click on 'Server Settings' and then on Server Settings screen, click on "General Settings".
Step 4: For creating a data source plugin in CloudFabrix OIA, Management Port needs to 8089 and enter value as shown below.
Addition of Splunk Enterprise as DataSource
Step 1: Login to cfxDimensions with user credentials.
Step 2: Under Data Sources App, select '+' on top right corner to add new Data Sources.
Step 3: Select 'Splunk' as Data Source Type as shown.
Enter the required details. The details required are
HostName: Splunk Enterprise System IP address
UserName: Username with admin rights
Password: Specify the password for the admin user
Port: API access port configured in previous section from Splunk Enterprise Instance (default: 8089)
TimeOut: The time required to wait for the response.
Note: Please chose https over http as http is not supported by Splunk.
Adding Splunk Enterprise Data Source in the Incident Room
Step 1: Login to cfxDimensions App and then click on OIA App.
Step 2: After creation of Data Source, add it under incident room creation.
Step 3: An incident gets created from Splunk Instance. You can also create new incidents.
Step 4: Click on Incident / ticket.
You data source widget chart (App Logs From Splunk) is as displayed below.
The fields (Timestamp-> time, Type->sourcetype, Message-> _raw)
Note: time, sourcetype, raw are Splunk index fields that are required to be present in the index.
Creating Alert Source for Splunk Enterprise
Step 1: Login to cfxDimensions and then select 'OIA app'.
Step 2: Select the Incident room where you want to add the alert source.
Step 3: Under Alerts tab, select Alert Sources on the right hand side corner as shown.
Step 4: Click on '+' to add new Alert Source as shown.
Step 5: Select Alert Source Type 'SplunkEnterprise' as shown.
Enter your cfxDimensions system DNS name or IP address as External IP address and 'inbound-alerts' at Topic field.
Click Save.
Step 6: From Alert Sources, select the hamburger menu on created source and click on View Details as shown.
The Details are as shown below.
Step 7: Copy the displayed URL in view details which follows the format "https://<Your cfxDimensions Platform IP>/webhooks/hookid/<uuid>/".
Step 8: Login to Splunk Enterprise Instance
Step 9: Go to 'Search and Reporting' section.
Step 10: Create new search from the pre-requisite index mentioned earlier in the document. Make sure you have the required field values mentioned. For example, Eg index = db_error_logs, host = *ip address*
Step 11: Save the search as alert
Step 12: While creating alert, the following 'Save As Alert' is displayed.
Select 'Webhook' from Actions drop down. Paste webhook url from earlier alert source. Enter alert name and any conditions.
Save the Alert.
Step 13: The alert can be viewed or edited from under the Alerts tab.
Step 14: It takes a while before alert is displayed in the OIA app which in turn creates it as an incident.
Step 15: These can be viewed in OIA app under alert tab.
Login as Customer Admin User into cfxDimensions App.
An example alert payload from Splunk is as shown below
Table describing the field mappings between Splunk Alert Payload and Alert Watch Fields.
Splunk Fields
Alert Watch Fields
Comments
Mandatory
Description
_time
raisedAt
YES
Time of the alert raised
host
assetId
YES
Asset Identifier
hostname
assetName
YES
Hostname of the asset
host
assetType
YES
Type of the Asset
ipaddress
assetIpAddress
YES
IP Address of the asset
sourcetype
alertType
YES
Type of the alert
sourcetype
alertCategory
YES
Category of the alert
sourcetype
componentId
YES
Component Identifier
source
componentName
YES
Name of the Component
severity
severity
YES
Alert Severity Ordered Number 0 - Critical, 1 - Major, 3 - Minor,
_raw
message
YES
Alert Payload
index
sourceId
YES
Identifier of the alert from the source system
Raised At : This field describes the time at which the alert was last occurred . Eg - The last occurred time on the incident ticket . (08/01/2020, 15:05:12)
Asset Id : This field uniquely Identifies an asset Eg (cfx-wpress-web01.demo.cloudfabrix.com) or Any UUID
Asset Name : This field describes the host name of the Device where alert is coming from, (Eg any FQDN (cfx-wpress-web01.demo.cloudfabrix.com)
Asset Type: The Type of the device that is raising alert .Here in this case ‘cfx-error-log’
Asset IP Address : The ip address of the device where the alert was raised form Eg (10.95.131.22)
Alert Category : The Type of the alert that device raises. In case of Splunk its same as the sourcetype of the log i.e ‘cfx-error-log’
Component Id : An unique identifier which identifies the sub component form the log messages. Eg In Logs the Source of the log messages
Component Name: The name of the Component (‘the path of the source log message’)
Severity: Alert Severity Ordered Number 0 - Critical, 1 - Major, 3 - Minor,
Message : This field describes the Log messages
SourceId : ‘The index form the splunk logs’
Last updated
