Home PC Games Linux Windows Database Network Programming Server Mobile  
           
  Home \ Linux \ LogStash log analysis display system     - Binder began to talk about inter-process communication mechanism from Android AIDL (Programming)

- Redis-2.8.17 installation and configuration process some errors (Linux)

- CentOS 6.5 installation using a data recovery software extundelete (Linux)

- CentOS 7 Docker build private warehouse registry (Linux)

- CentOS 6.5_x64 install Oracle 11g R2 (Database)

- Use LVM partition on Ubuntu and easily adjust the snapshot (Linux)

- 30 Practical Linux system administrators will learn the command (Linux)

- Oracle 11g upgrade PSU detailed steps (Database)

- Linux performance monitoring - CPU, Memory, IO, Network (Linux)

- Oracle LONG RAW BLOB CLOB type of presentation (Database)

- Java Access Control (Programming)

- Systemd on RHEL7 (Linux)

- Ubuntu 14.10 / 14.04 / 12.04 How to install Kodi 14.0 RC3 (Linux)

- Git / Github use notes (Linux)

- MySQL server after an unexpected power outage can not start (Database)

- C ++ free store and heap (Programming)

- Oracle 10g after the failure to start the upgrade (Oracle instance terminated. Disconnection forced) (Database)

- In Debian 4.3 compiler under Linux-2.6.28 kernel Summary (Programming)

- Intel Graphics Installer 1.0.3 released, support for Ubuntu (Linux)

- Upgrading from Fedora 20 to 21 (Linux)

 
         
  LogStash log analysis display system
     
  Add Date : 2017-08-30      
         
         
         
  Introduction

Log management is usually a gradual breakdown - when the log for the most important time, that is, when problems arise, the gradual process began.
Log management will generally go through three stages:

The primary administrator will examine the logs through some traditional tools such as cat, tail, sed, awk, perl, and grep, but it is limited to a small number of hosts and log file types;
Taking into account the scalability of the real problem, log management will be gradually evolved, such as rsyslog and syslog-ng tools such as centralized management;
As log information grows, it becomes more difficult to extract the information needed from the fast growing log data stream and associate it with other associated events, and LogStash provides A good solution
Advantages of LogStash:
Better parsing capabilities for log data;

More flexible log storage

With search and directory functions

Easy to install, scalable, good performance

Design and architecture
LogStash is written in the JRuby language, based on a message-based simple architecture and runs on the Java Virtual Machine (JVM). Unlike a separate agent or server, LogStash can be configured with a single agent to be combined with other open source software for different functions.
In the LogStash ecosystem, there are four major components:
Shipper: send events to the LogStash; usually, the remote agent (agent) only need to run this component can;

Broker and Indexer: receive and index the event;

Search and Storage: allows the search and storage of events;

Web Interface: Web-based display interface

It is because of the above components in the LogStash architecture can be independently deployed, it provides a better cluster scalability.

In most cases, LogStash hosts fall into two broad categories:

Agent host (agent host): as the event transporter (shipper), the various log data sent to the central host; just run the Logstash agent (agent) program;

Central host: Runs various components, including intermediate brokers, indexers, search and storage, and Web interfaces, to enable the logging of data Of the receiving, processing and storage.

deploy
Basic environment
Yum install java-1.7.0-openjdk
Java -version # Ensure java version 1.7
Deploy LogStash
# Download
Wget https://download.elasticsearch.org/logstash/logstash/logstash-1.3.1-flatjar.jar -Ologstash.jar
# start up
Java - jar logstash.jar agent - v - f shipper.conf # start shipper
Java -jar logstash.jar agent -v-f indexer.conf # Start indexer
  
Deploy Redis
# Installation
Yum install redis-server
# start up
/etc/init.d/redis-server start
# Test
$ Redis-cli -h 192.168.12.24
Redis 192.168.12.24:6379> PING
PONG
Deploy Elasticsearch
# Download
Wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.8.noarch.rpm
# Installation
Rpm -ivh elasticsearch-0.90.8.noarch.rpm
# start up
/etc/init.d/elasticsearch status
Start Kibana
# Installation
Java -jar logstash.jar web # LogStash 1.3.1 comes with Kibana
# View
Http://192.168.12.24:9292

Logstash configuration files and plug-ins
Input {
 Stdin {}
 File {
  Type => "syslog"
  Path => [ "/ var / log / secure", "/ var / log / messages"]
  Exclude => [ "* .gz", "shipper.log"]
 }}
    Zeromq {
      Address => [ "tcp: //192.168.8.145: 8889"]
  Mode => "client"
      Type => "zmq-input"
      Topic => "weblog"
      Topology => "pubsub"
  Codec => "json"
    }}
  }}
  Filter {
 Mutate {
  Gsub => [ "message", "APPS weblog", ""]
  Gsub => [ "message", "{", ""]
  Gsub => [ "message", "}", ""]
 }}
  }}
  Output {
 Stdout {debug => true debug_format => "json"}
    
 Elasticsearch {
      Cluster => "logstash"
  Codec => "json"
    }}
}}

Log categories and processing methods
Apache log: custom apache output log format, json output, no filter to participate

Postfix log: can not be customized, need to use filters such as grok filter

Tomcat log: the need to merge multiple logs into an event, and exclude blank lines

Cluster extension
Extended architecture

Precautions
Redis: deploy multiple, only available high-availability role, no load sharing role, you can use ZeroMQ instead
ElasticSearch:
# Check the node status:
 Curl -XGET 'http://127.0.0.1:9200/_cluster/health?pretty=true'
   Green status: All shards are allocated and are operating normally
   Yellow status: Only the primary shard is allocated, such as when the cluster is replicating data between nodes
   Red status: There is a shard not allocated
# Cluster monitoring:
 Paramedic Tools:
   Install: / usr / share / elasticsearch / bin / plugin -install karmi / elasticsearch-paramedic
   See: http://log.linuxidc.net: 9200 / _plugin / paramedic / index.html
 Bigdesk Tools:
   Install: / usr / share / elasticsearch / bin / plugin -install lukas-vlcek / bigdesk
   See: http://log.linuxidc.net: 9200 / _plugin / bigdesk / index.html
   
# Data Retention Policy:
   1.LogStash default to create an index for each day, you can manually delete the index
    Curl -XDELETE http://127.0.0.1:9200/logstash-2013.12.19
    Shell optimization script: https://github.com/cnf/logstash-tools/blob/master/elasticsearch/clean-elasticsearch.sh
   2. Optimization index:
    Curl -XPOST 'http://127.0.0.1:9200/logstash-2013.12.19/_optimize'
    Curl -XPOST 'http://127.0.0.1:9200/_optimize' # optimize all the index
    Curl 'http://127.0.0.1:9200/logstash-2013.12.19/_stats?clear=true&store=true&pretty=true' # Check the index size, index too much will affect the optimization time-consuming
   3. Default index data directory: / var / lib / elasticsearch / logstash

References
LogStash official website: http: //www.logstash.net/
Elasticsearch official website: http: //www.elasticsearch.org/
Kibana Query Syntax: http: //lucene.apache.org/core/3_6_1/queryparsersyntax.html
     
         
         
         
  More:      
 
- JavaScript common array manipulation functions and usage (Programming)
- Modifying the system registry protection server security (Linux)
- Ubuntu program using the Sound Recorder (Linux)
- Linux Monitoring Command (Linux)
- Talk Packages (Linux)
- Modify MySQL character encoding under Linux (Database)
- Compare several ways to transfer files between Linux hosts (Linux)
- Ubuntu 15.04 and CentOS 7 to deploy .NET, Mono and DNX (Server)
- Recycle Bin function realization in Linux (Linux)
- Hadoop1.2.1 plug compilation (Server)
- Linux see whether there is a hacker program (Linux)
- The wrong in Linux: too many open files (Linux)
- Ansible module Know (Linux)
- MySQL multi-master-slave synchronization (Database)
- A new method for Linux hidden files (Linux)
- linux raid levels and concepts introduced (Linux)
- To compile and install Clang 3.5.0 in Linux CentOS (Linux)
- Modify Linux SSH default port 22 in several ways (Linux)
- Redhat 5 prohibit IPv6 (Linux)
- View and modify Linux machine name (Linux)
     
           
     
  CopyRight 2002-2022 newfreesoft.com, All Rights Reserved.