CloudFabrix Documentation
Asset IntelligenceOps IntelligenceObservabilityRobotic Data
  • Getting Started
  • CloudFabrix Overview
  • AIOps Platform
  • AIOps Solutions
  • CloudFabrix RDA
    • RDA - Overview
      • RDA - Terminology and Artifacts
    • RDA - Installation
      • Linux OS
      • Windows OS
      • Mac OS
      • RDA Client
      • Worker Nodes
      • Event Gateway
      • Edge Collector
      • Log Shippers
        • Filebeat
        • Fluentd
        • Rsyslog
        • Syslog (udp)
        • Splunk forwarder (Windows and Linux)
        • Winlogbeat (Windows)
      • RDA Log Archives
    • RDA - Administration
      • RDA - Configuration
    • RDA - User Guide
      • RDA - AIOps Studio
        • AIOps Studio - Solution Packages
        • AIOps Studio - Pipelines
        • AIOps Studio - Explore
        • AIOps Studio - Administration
        • RDA CLI in UI
        • AIOps Studio - Examples
          • File Operations
          • Loop Operations
          • Data Management Operations - cfxdm
          • Data mapping - cfxdm - dm:eval
          • Filters - cfxdm - dm:filter
          • Data mapping - cfxdm - dm:map
          • Metadata - cfxdm-dm:metadata
          • Data mapping - cfxdm - dm:functions
            • Any_non_null
            • Concat
            • Datetime
            • Date and Timestamp
            • Evaluate
            • Fixed
            • Highest
            • Join
            • jsonDecode
            • Lower
            • Lowest
            • Match
            • Minutes_Between
            • Replace
            • Seconds_Between
            • Slice
            • Split
            • Strip
            • To_Numeric
            • Ts_To_Datetimestr
            • Upper
            • When_Null
          • Data Mapping cfxdm - dm:sort
          • Data Mapping cfxdm - dm:head
          • Data Mapping cfxdm - dm: tail
          • Data Mapping cfxdm - dm: dedup
          • Data Mapping cfxdm - dm:selectcolumns
          • Data Mapping cfxdm - dm:fixcolumns
          • Data Mapping cfxdm - dm:mergecolumns
          • Data Mapping cfxdm - dm:describe
          • Data Mapping cfxdm - dm:save
          • Data Mapping cfxdm - dm:savedlist
          • Data Mapping cfxdm - dm:recall
          • Data Mapping cfxdm - dm:concat
          • Data Mapping cfxdm - dm:groupby
          • Data Mapping cfxdm - dm:to_type
          • Data Mapping cfxdm - dm:enrich
          • Data Mapping cfxdm - dm:dns_ip_to_name
          • Data Mapping cfxdm - dm:dns_name_to_ip
        • AIOps Studio - Datasource Examples
          • Elasticsearch (v1)
      • RDA - Data Management (cfxdm)
        • cfxdm - dm:filter
        • cfxdm - dm:map
        • cfxdm - dm:functions
        • cfxdm - dm:sort
        • cfxdm - dm:head
        • cfxdm - dm:tail
        • cfxdm - dm:dedup
        • cfxdm - dm:selectcolumns
        • cfxdm - dm:mergecolumns
        • cfxdm - dm:describe
        • cfxdm - dm:hist
        • cfxdm - dm:bin
        • cfxdm - dm:fixcolumns
        • cfxdm - dm:save
        • cfxdm - dm:savedlist
        • cfxdx - dm:recall
        • cfxdm - dm:concat
        • cfxdm - dm:groupby
        • cfxdm - dm:enrich
        • cfxdm - dm:to_type
        • cfxdm - dm:dns_ip_to_name
        • cfxdm - dm:dns_name_to_ip
        • cfxdm - files:loadfile
      • RDA Terminal
        • Examples using Terminal / Commandline
    • RDA - Bot Documentation
    • RDA - Datasource Integrations
      • AppDynamics
      • Dynatrace
      • Dell EMC Unity
      • Elasticsearch
      • Infoblox NetMRI
      • Kubernetes Cluster
      • Linux OS
      • Microsoft Windows Server OS
      • Nagios XI
      • NetApp Clustered ONTAP
      • PRTG Network Monitor
      • VMware vCenter
      • VMware vRealize Operations
    • RDA - Python API
      • Class CaaSDataset
      • Class CaaSClient
      • Python API Example
    • RDA - FAQ
      • Download and Installation
      • Troubleshooting
  • CFXQL - CFX Query Language
    • CFXQL User Interface
  • Operations Intelligence Analytics (OIA)
    • Solution Overview
      • Navigating cfxOIA
      • Any title
    • Administration Guide
      • Active Directory Integration
      • Add Customer
      • First Steps
      • OIA Users
      • Add New Users and Assign Roles
      • Add Environment
      • Add Gateway
      • Add DataSource
      • Add Project
      • Project Configuration
      • Stacks
      • Teams
    • User Guide
      • Incidents
        • Incident
        • Stack
      • Alerts
      • Outcomes
  • INTEGRATIONS
    • Zabbix
    • AppDynamics
    • Dynatrace
    • NetApp C Mode
    • NetApp 7 Mode
    • Splunk Enterprise
    • VMware vCenter
  • OBSERVABILITY
  • Observability - IT Infrastructure Monitoring (cfxPulse)
    • Getting Started
    • Solution Overview
    • Installing cfxPulse Collector
    • Administrator Quick Start Guide
      • Prerequisites
      • Accessing cfxPulse
      • Configuration of cfxPulse
      • Setting Up Monitoring
      • Monitoring Using Prometheus Agents
      • How to add Prometheus Agent Details
      • Discovery of Devices
      • Monitoring Dashboard
    • End User Quick Start Guide
      • Portal Navigation
      • Monitoring Analysis
      • Monitoring
      • Alerts and Incidents
      • Reports
      • NOC/Ops
      • Configuration Backups
      • Interacting With Tabular Reports
      • Creating Custom Dashboards
  • Observability - Log Monitoring & Analytics (CFX LogAnalytics or CLA)
    • Getting Started
    • Solution Overview
    • Solution Key Components
    • Log Forwarding
      • Install and Configure Logstash
      • Sending Logs to Logstash Forwarder
    • Log Collection
      • Collecting Logs from Linux
    • Log Transformation & Enrichment
    • Logstash Installation
      • How to install Java / Logstash on client side
  • Asset Intelligence Analytics (AIA) Solution
    • Getting Started
    • Solution Overview
    • AIA Roles
      • Platform Admin
        • Managed Service Provider (MSP)
        • Authentication Server
        • Set Up Services
        • Organizations
        • Users
      • Organization Admin
        • My Organizations
      • Organization Executive
      • Organization User
    • AIA Tasks, Functions
      • Home Page Navigation
      • Filters
      • Settings Menu
      • Notifications
      • Authentical Server
      • How to Add, Edit, Delete MSP
      • Actions
        • Services
        • Files
        • Dictionaries
        • Discovery Jobs
        • Snapshots
        • Clambda Jobs
        • State Operations
        • Replacement Rate
      • Details
        • Overall
        • POR Insights
        • HW Assets
        • SW Assets
        • Contracts
        • App Dependency
        • Asset List
    • AIA API
    • Enterprise Discovery
      • cfxEdgeCollector
        • Deployment of cfxEdgeCollector
        • Configuration of cfxEdgeCollector
        • cfxEdgeCollector Command Line Options
        • cfxEdgeCollector Help Command
        • Working With cfxEdgeCollector
        • cfxEdgeCollector Auto Export
    • Asset Intelligence & Analytics (AIA) (Delete)
  • CloudFabrix SaaS
    • Signup
    • Navigation
    • User Roles
  • Support
    • Contact Support
Powered by GitBook
On this page
  • Prerequisites
  • Installation
  • Basic Configuration
  • Configuration File
  1. Observability - Log Monitoring & Analytics (CFX LogAnalytics or CLA)
  2. Log Forwarding

Install and Configure Logstash

Install and Configure LogStash for Log Forwarding and Aggregation

PreviousLog ForwardingNextSending Logs to Logstash Forwarder

Last updated 4 years ago

Logstash can serve as local on-premise log aggregator and forwarder that can send logs to DLA running in CloudFabrix SaaS environment. Following are instructions for installing Logstash in customer’s environment.

Prerequisites

Java -

Logstash Version - 2.x / 5.x / 6.x / 7.x

Software

Version

Java

Java version 8 or Java Version 11. Ensure JAVA_HOME environment variable is set.

Linux

CentOS 7.x

Debian 8.x

Ubuntu 18.0.4

Installation

Detailed installation for most popular Linux distributions are provided in the . Abridged version of installation instructions for some common distributions are provided here.

Basic Configuration

The following example tails the /var/log/messages file and forwards every line to your Logs App.To start pushing logs, you must create a file named /etc/logstash/conf.d/logsene.conf with the below text and restart Logstash. An example is as shown below.

input {
  file {
    path => "/var/log/messages"
    start_position => "beginning"
  }
}

output {
  elasticsearch {
    # use port 80 for plain HTTP, instead of HTTPS
    hosts => "logsene-receiver.cfxDLA.com:443"
    # set to false if you don't want to use SSL/HTTPS
    ssl => "true"
    index => "db0461c5-7106-4b3e-b2af-42f20fd95b0f"
    manage_template => false
  }
}

Configuration File

Logstash Configuration file (logstash-sample.conf) will be located at /etc/logstash/ folder where logstash is installed which looks like below. Once the file is updated it is copied to /etc/logstash/conf.d folder as configured in pipelines.yml file (which is located in /etc/logstash). Logstash configuration file will contain three objects.

Filebeat installation and configuration covered in detail at 'Install and Configure Filebeat'.

  • Input : Logstash input is about the information of the filebeat IP address and port.

input {
  beats {
        client_inactivity_timeout => "300"
        host => "<FileBeat IP Address>"
        port => "<FileBeat Port>"
     }
}

Note :- “client_inactivity_timeout” time is in ms. FileBeat ip address should be given in the “host” field and FileBeat Port should be given in the “port” field.

  • Filter: Filter section is about the Grok Patterns of the filebeat log file. Grok Pattern needs to be specific to the log file which you had given in the filebeat.yml input file

filter {
 grok {
    match => { "message" => ["<Grok Pattern of the log>"]}
 }
}
  • Output :- Output section is about the filebeat output where the log files should be transferred to. An example is shown. Change IP addresses.

 output {
        kafka {
        codec => json
        topic_id => "<Kafka Topic Id>"
        bootstrap_servers => "ip address:9093,ip address:9093,ip address:9093"
        security_protocol => "SSL"
        ssl_endpoint_identification_algorithm => " "
        ssl_truststore_location => "<Location of Truststore"
        ssl_truststore_password => "<Password of the Truststore>"
        }
 }

Note :- “ssl_endpoint_identification_algorithm” name should be the machine hostname where ssl truststore is generated.

official website
How to install Java / Logstash on client side