Collecting Logs from Linux

Filebeat is the recommended way to ship log files to CLA.

Prerequisites:

• Filebeat 6.3 (or higher) installed on your system. Please see the installation instructions

• Allow outgoing traffic to destination port 5044 for Logstash and port 9200 for elastic search.

• Either port 5044 or 9200 should be open not both.

Installation on Linux / Mac

Step 1: Download the certificate

wget https://raw.githubusercontent.com/cfxDLAio/public-certificates/master/COMODORSADomainValidationSecureServerCA.crt

Step 2: Copy the certificate to desired location

sudo mkdir -p /etc/pki/tls/certs
sudo cp COMODORSADomainValidationSecureServerCA.crt /etc/pki/tls/certs/

Step 3: Configure Filebeat

On Linux file systems, the configuration should be placed user

/etc/filebeat/filebeat.yml

Either you can use configuration wizard to create the configuration file or you can manually edit the configuration file. Always make sure that you have taken backup of original configuration file.

The file contains two main sections:

• prospectors: define the files that needs to be monitored

• outputs: responsible for sending the data in JSON format

Note 1:

There is a difference when configuring a prospector that harvests JSON files vs a prospector that harvests simple text line logs. Please follow the following guidelines.

When harvesting JSON files, please add cfxDLAio_codec equals to json to the fields object.

When harvesting text files, please add cfxDLAio_codec equals to plain to the fields object.

Note 2:

The fields_under_root property should always set to be true.

Note 3:

The type property should be defined, it is used to identify the type of log data and provide you with a richer logging experience. For complete list of known types, please click here. If the type you are looking for is not listed, please select the type that is closer or best describes your data.

The sample configuration file will look like the following:

filebeat.inputs:
- type: log
  paths: 
  - /path/to/log/file.log
  fields:
    cfxDLAio_codec: plain
    token: GsfEyZvknXZFtrNfqdmrmiJXvYIKJbID
    type: MY_LOG_TYPE
  fields_under_root: true
  encoding: utf-8
  ignore_older: 3h
  
- type: log
  paths: 
  - /path/to/json/file.json
  fields:
    cfxDLAio_codec: json
    token: GsfEyZvknXZFtrNfqdmrmiJXvYIKJbID
    type: MY_JSON_LOG_TYPE
  fields_under_root: true
  encoding: utf-8
  ignore_older: 3h
  
#For version 6.x and lower uncomment the line below and remove the line after it 
#filebeat.registry_file: /var/lib/filebeat/registry 
 
filebeat.registry.path: /var/lib/filebeat
 
#The following processors are to ensure compatibility with version 7
processors:
- rename:
    fields:
     - from: "agent"
       to: "beat_agent"
    ignore_missing: true
- rename:
    fields:
     - from: "log.file.path"
       to: "source"
    ignore_missing: true
    
############################# Output ##########################################
output:
  logstash:
    hosts: ["listener-ca.cfxdla.io:5044"]
    ssl:
      certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']

You can add as many file entries as you like. All file entries should contain the cfxDLA token.