Splunk Enterprise

Prerequisites:
1.Splunk Enterprise Version 8.0.0
2.Splunk Indexes (Please refer Splunk Documentation on how to create indexes)
a. Eg: Webserver host logs created with log index  “web_error_logs” with field values hostname, severity, ipaddress etc...
b. DB Server host logs created with log index “db_error_logs” with field values hostname, severity, ipaddress etc...
Note-1: For field “severity”, you can extract from the log message (Eg: info, error, warn, debug etc)
Note-2: Fields “hostname” and “ipaddress” should be the HostName and IP address of the host on which Splunk agent is installed and collecting the OS or Application logs.
To add the above custom fields on Splunk Agent follow the below steps (Linux version).
1
cd /opt/splunkforwarder/etc/system/local
2
Edit inputs.conf
3
Add the below line to add additional fields
4
_meta = key1::value1 key2::value2
5
Eg: _meta = ipaddress::10.95.131.23 hostname::cfx-wpress-db01.demo.cloudfabrix.com
6
Restart splunk agent
7
/opt/splunkforwarder/bin/splunk stop
8
/opt/splunkforwarder/bin/splunk start
Copied!
          3. Enable API port 8089 on Splunk Enterprise administration UI
Enabling API Port 8089
Step 1: Log into Your Splunk Enterprise Instance with admin credentials
Step 2: From Top Menu bar, click on 'Settings' tab.
Step 3: Click on 'Server Settings' and then on Server Settings screen, click on "General Settings".
Step 4: For creating a data source plugin in CloudFabrix OIA, Management Port needs to 8089 and enter value as shown below.
Management Port = 8089
Addition of Splunk Enterprise as DataSource
Step 1: Login to cfxDimensions with user credentials.
Step 2: Under Data Sources App, select '+' on top right corner to add new Data Sources.
Add new data source
Step 3: Select 'Splunk' as Data Source Type as shown.
​
Splunk as Data Source
Enter the required details. The details required are
HostName: Splunk Enterprise System IP address
UserName: Username with admin rights
Password: Specify the password for the admin user
Port: API access port configured in previous section from Splunk Enterprise Instance (default: 8089)
TimeOut: The time required to wait for the response.
Note: Please chose https over http as http is not supported by Splunk.
Adding Splunk Enterprise Data Source in the Incident Room
Step 1: Login to cfxDimensions App and then click on OIA App.
Step 2: After creation of Data Source, add it under incident room creation.
Add Project
Step 3: An incident gets created from Splunk Instance. You can also create new incidents.
Splunk Incident
Step 4: Click on Incident / ticket.
You data source widget chart (App Logs From Splunk) is as displayed below.
The fields (Timestamp-> time, Type->sourcetype, Message-> _raw)
Note: time, sourcetype, raw are Splunk index fields that are required to be present in the index.
Creating Alert Source for Splunk Enterprise
Step 1: Login to cfxDimensions and then select 'OIA app'.
Step 2: Select the Incident room where you want to add the alert source.
Step 3: Under Alerts tab, select Alert Sources on the right hand side corner as shown.
Select Alert Sources
Step 4: Click on '+' to add new Alert Source as shown.
Click '+' to add new alert source
Step 5: Select Alert Source Type 'SplunkEnterprise' as shown.
Selecting Splunk Enterprise as Data Source
Enter your cfxDimensions system DNS name or IP address as External IP address and 'inbound-alerts' at Topic field.
Click Save.
Step 6: From Alert Sources, select the hamburger menu on created source  and click on View Details as shown.
The Details are as shown below.
Step 7: Copy the displayed URL in view details which follows the format "https://<Your cfxDimensions Platform IP>/webhooks/hookid/<uuid>/".
Step 8: Login to Splunk Enterprise Instance
Step 9: Go to 'Search and Reporting' section.
Step 10: Create new search from the pre-requisite index mentioned earlier in the document. Make sure you have the required field values mentioned. For example, Eg index = db_error_logs, host = *ip address*
Step 11: Save the search as alert 
Step 12: While creating alert, the following 'Save As Alert' is displayed.
Select 'Webhook' from Actions drop down. Paste webhook url from earlier alert source. Enter alert name and any conditions.
Save the Alert.
Step 13: The alert can be viewed or edited from under the Alerts tab.
Note: More information on Splunk alerts can be read here (https://docs.splunk.com/Documentation/SplunkCloud/8.0.1/Alert/Aboutalerts).
Step 14: It takes a while before alert is displayed in the OIA app which in turn creates it as an incident.
Step 15: These can be viewed in OIA app under alert tab.
Login as Customer Admin User into cfxDimensions App.
An example alert payload from Splunk is as shown below
1
{
2
  "search_name": "web logs alerts",
3
  "owner": "admin",
4
  "sid": "rt_scheduler__admin__search__RMD549d1c662f912271f_at_1577189234_887.781204",
5
  "result": {
6
    "_sourcetype": "cfx-error-log",
7
    "_time": "1578399544.002748",
8
    "date_mday": "7",
9
    "_kv": "1",
10
    "punct": "[___::._]_[:]_[_]_[_...:]__'/////.'______",
11
    "severity": "error",
12
    "_serial": "1815613",
13
    "index": "web_error_logs",
14
    "linecount": "",
15
    "sourcetype": "cfx-error-log",
16
    "date_month": "january",
17
    "_confstr": "source::/var/log/httpd/cfx-error-log|host::cfx-wpress-web01|10.95.131.22|cfx-error-log",
18
    "ipaddress": "10.95.131.22",
19
    "source": "/var/log/httpd/cfx-error-log",
20
    "timeendpos": "32",
21
    "_eventtype_color": "",
22
    "date_minute": "19",
23
    "pid_id": "16494",
24
    "splunk_server": "splunk-vm01",
25
    "hostname": "cfx-wpress-web01.demo.cloudfabrix.com",
26
    "_raw": "[Tue Jan 07 04:19:04.002748 2020] [:error] [pid 16494] [client 10.95.122.127:43968] script '/var/www/html/wordpress/myapp323.php' not found or unable to stat",
27
    "date_second": "4",
28
    "date_zone": "local",
29
"_subsecond": ".002748",
30
    "date_wday": "tuesday",
31
    "host": "cfx-wpress-web01|10.95.131.22",
32
    "eventtype": "",
33
    "timestartpos": "5",
34
    "_indextime": "1578399544",
35
    "date_hour": "4",
36
    "date_year": "2020",
37
    "_si": [
38
      "splunk-vm01",
39
      "web_error_logs"
40
    ]
41
  },
42
  "results_link": "http://splunk-vm01:8000/app/search/@go?sid=rt_scheduler__admin__search__RMD549d1c662f912271f_at_1577189234_887.781204",
43
  "app": "search"
44
}
45
​
Copied!
Table describing the field mappings between Splunk Alert Payload and Alert Watch Fields.
​

Splunk Fields
Alert Watch Fields
Comments
Mandatory
Description
_time
raisedAt
​
YES
Time of the alert raised
host
assetId
​
YES
Asset Identifier
hostname
assetName
​
YES
Hostname of the asset
host
assetType
​
YES
Type of the Asset
ipaddress
assetIpAddress
​
YES
IP Address of the asset
sourcetype
alertType
​
YES
Type of the alert
sourcetype
alertCategory
​
YES
Category of the alert
sourcetype
componentId
​
YES
Component Identifier
source
componentName
​
YES
Name of the Component
severity
severity
​
YES
Alert Severity Ordered Number
0 - Critical, 1 - Major, 3 - Minor, 
_raw
message
​
YES
Alert Payload
index
sourceId
​
YES
Identifier of the alert from the source system
​
1.Raised At  : This field describes the time at which the alert was last occurred . Eg - The last occurred time on the incident ticket . (08/01/2020, 15:05:12)
2.Asset Id : This field uniquely Identifies an asset  Eg (cfx-wpress-web01.demo.cloudfabrix.com) or Any UUID
3.Asset Name : This field describes the host name of the Device where alert is coming from,  (Eg any FQDN (cfx-wpress-web01.demo.cloudfabrix.com)
4.Asset Type: The Type of the device that is raising alert .Here in this case ‘cfx-error-log’
5.Asset IP Address : The ip address of the device where the alert was raised form Eg (10.95.131.22)
6.Alert Category : The Type of the alert that device raises. In case of Splunk its same as the sourcetype of the log i.e ‘cfx-error-log’
7.Component Id : An unique identifier which identifies the sub component form the log messages. Eg In Logs the Source of the log messages
8.Component Name: The name of the Component (‘the path of the source log message’)
9.Severity: Alert Severity Ordered Number 0 - Critical, 1 - Major, 3 - Minor, 
10.Message : This field describes the Log messages
11.SourceId : ‘The index form the splunk logs’