Home PC Games Linux Windows Database Network Programming Server Mobile  
  Home \ Linux \ LogStash log analysis display system     - Binder began to talk about inter-process communication mechanism from Android AIDL (Programming)

- Redis-2.8.17 installation and configuration process some errors (Linux)

- CentOS 6.5 installation using a data recovery software extundelete (Linux)

- CentOS 7 Docker build private warehouse registry (Linux)

- CentOS 6.5_x64 install Oracle 11g R2 (Database)

- Use LVM partition on Ubuntu and easily adjust the snapshot (Linux)

- 30 Practical Linux system administrators will learn the command (Linux)

- Oracle 11g upgrade PSU detailed steps (Database)

- Linux performance monitoring - CPU, Memory, IO, Network (Linux)

- Oracle LONG RAW BLOB CLOB type of presentation (Database)

- Java Access Control (Programming)

- Systemd on RHEL7 (Linux)

- Ubuntu 14.10 / 14.04 / 12.04 How to install Kodi 14.0 RC3 (Linux)

- Git / Github use notes (Linux)

- MySQL server after an unexpected power outage can not start (Database)

- C ++ free store and heap (Programming)

- Oracle 10g after the failure to start the upgrade (Oracle instance terminated. Disconnection forced) (Database)

- In Debian 4.3 compiler under Linux-2.6.28 kernel Summary (Programming)

- Intel Graphics Installer 1.0.3 released, support for Ubuntu (Linux)

- Upgrading from Fedora 20 to 21 (Linux)

  LogStash log analysis display system
  Add Date : 2017-08-30      

Log management is usually a gradual breakdown - when the log for the most important time, that is, when problems arise, the gradual process began.
Log management will generally go through three stages:

The primary administrator will examine the logs through some traditional tools such as cat, tail, sed, awk, perl, and grep, but it is limited to a small number of hosts and log file types;
Taking into account the scalability of the real problem, log management will be gradually evolved, such as rsyslog and syslog-ng tools such as centralized management;
As log information grows, it becomes more difficult to extract the information needed from the fast growing log data stream and associate it with other associated events, and LogStash provides A good solution
Advantages of LogStash:
Better parsing capabilities for log data;

More flexible log storage

With search and directory functions

Easy to install, scalable, good performance

Design and architecture
LogStash is written in the JRuby language, based on a message-based simple architecture and runs on the Java Virtual Machine (JVM). Unlike a separate agent or server, LogStash can be configured with a single agent to be combined with other open source software for different functions.
In the LogStash ecosystem, there are four major components:
Shipper: send events to the LogStash; usually, the remote agent (agent) only need to run this component can;

Broker and Indexer: receive and index the event;

Search and Storage: allows the search and storage of events;

Web Interface: Web-based display interface

It is because of the above components in the LogStash architecture can be independently deployed, it provides a better cluster scalability.

In most cases, LogStash hosts fall into two broad categories:

Agent host (agent host): as the event transporter (shipper), the various log data sent to the central host; just run the Logstash agent (agent) program;

Central host: Runs various components, including intermediate brokers, indexers, search and storage, and Web interfaces, to enable the logging of data Of the receiving, processing and storage.

Basic environment
Yum install java-1.7.0-openjdk
Java -version # Ensure java version 1.7
Deploy LogStash
# Download
Wget https://download.elasticsearch.org/logstash/logstash/logstash-1.3.1-flatjar.jar -Ologstash.jar
# start up
Java - jar logstash.jar agent - v - f shipper.conf # start shipper
Java -jar logstash.jar agent -v-f indexer.conf # Start indexer
Deploy Redis
# Installation
Yum install redis-server
# start up
/etc/init.d/redis-server start
# Test
$ Redis-cli -h
Redis> PING
Deploy Elasticsearch
# Download
Wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.8.noarch.rpm
# Installation
Rpm -ivh elasticsearch-0.90.8.noarch.rpm
# start up
/etc/init.d/elasticsearch status
Start Kibana
# Installation
Java -jar logstash.jar web # LogStash 1.3.1 comes with Kibana
# View

Logstash configuration files and plug-ins
Input {
 Stdin {}
 File {
  Type => "syslog"
  Path => [ "/ var / log / secure", "/ var / log / messages"]
  Exclude => [ "* .gz", "shipper.log"]
    Zeromq {
      Address => [ "tcp: // 8889"]
  Mode => "client"
      Type => "zmq-input"
      Topic => "weblog"
      Topology => "pubsub"
  Codec => "json"
  Filter {
 Mutate {
  Gsub => [ "message", "APPS weblog", ""]
  Gsub => [ "message", "{", ""]
  Gsub => [ "message", "}", ""]
  Output {
 Stdout {debug => true debug_format => "json"}
 Elasticsearch {
      Cluster => "logstash"
  Codec => "json"

Log categories and processing methods
Apache log: custom apache output log format, json output, no filter to participate

Postfix log: can not be customized, need to use filters such as grok filter

Tomcat log: the need to merge multiple logs into an event, and exclude blank lines

Cluster extension
Extended architecture

Redis: deploy multiple, only available high-availability role, no load sharing role, you can use ZeroMQ instead
# Check the node status:
 Curl -XGET ''
   Green status: All shards are allocated and are operating normally
   Yellow status: Only the primary shard is allocated, such as when the cluster is replicating data between nodes
   Red status: There is a shard not allocated
# Cluster monitoring:
 Paramedic Tools:
   Install: / usr / share / elasticsearch / bin / plugin -install karmi / elasticsearch-paramedic
   See: http://log.linuxidc.net: 9200 / _plugin / paramedic / index.html
 Bigdesk Tools:
   Install: / usr / share / elasticsearch / bin / plugin -install lukas-vlcek / bigdesk
   See: http://log.linuxidc.net: 9200 / _plugin / bigdesk / index.html
# Data Retention Policy:
   1.LogStash default to create an index for each day, you can manually delete the index
    Curl -XDELETE
    Shell optimization script: https://github.com/cnf/logstash-tools/blob/master/elasticsearch/clean-elasticsearch.sh
   2. Optimization index:
    Curl -XPOST ''
    Curl -XPOST '' # optimize all the index
    Curl '' # Check the index size, index too much will affect the optimization time-consuming
   3. Default index data directory: / var / lib / elasticsearch / logstash

LogStash official website: http: //www.logstash.net/
Elasticsearch official website: http: //www.elasticsearch.org/
Kibana Query Syntax: http: //lucene.apache.org/core/3_6_1/queryparsersyntax.html
- JavaScript common array manipulation functions and usage (Programming)
- Modifying the system registry protection server security (Linux)
- Ubuntu program using the Sound Recorder (Linux)
- Linux Monitoring Command (Linux)
- Talk Packages (Linux)
- Modify MySQL character encoding under Linux (Database)
- Compare several ways to transfer files between Linux hosts (Linux)
- Ubuntu 15.04 and CentOS 7 to deploy .NET, Mono and DNX (Server)
- Recycle Bin function realization in Linux (Linux)
- Hadoop1.2.1 plug compilation (Server)
- Linux see whether there is a hacker program (Linux)
- The wrong in Linux: too many open files (Linux)
- Ansible module Know (Linux)
- MySQL multi-master-slave synchronization (Database)
- A new method for Linux hidden files (Linux)
- linux raid levels and concepts introduced (Linux)
- To compile and install Clang 3.5.0 in Linux CentOS (Linux)
- Modify Linux SSH default port 22 in several ways (Linux)
- Redhat 5 prohibit IPv6 (Linux)
- View and modify Linux machine name (Linux)
  CopyRight 2002-2022 newfreesoft.com, All Rights Reserved.