| |
Introduction
Log management is usually a gradual breakdown - when the log for the most important time, that is, when problems arise, the gradual process began.
Log management will generally go through three stages:
The primary administrator will examine the logs through some traditional tools such as cat, tail, sed, awk, perl, and grep, but it is limited to a small number of hosts and log file types;
Taking into account the scalability of the real problem, log management will be gradually evolved, such as rsyslog and syslog-ng tools such as centralized management;
As log information grows, it becomes more difficult to extract the information needed from the fast growing log data stream and associate it with other associated events, and LogStash provides A good solution
Advantages of LogStash:
Better parsing capabilities for log data;
More flexible log storage
With search and directory functions
Easy to install, scalable, good performance
Design and architecture
LogStash is written in the JRuby language, based on a message-based simple architecture and runs on the Java Virtual Machine (JVM). Unlike a separate agent or server, LogStash can be configured with a single agent to be combined with other open source software for different functions.
In the LogStash ecosystem, there are four major components:
Shipper: send events to the LogStash; usually, the remote agent (agent) only need to run this component can;
Broker and Indexer: receive and index the event;
Search and Storage: allows the search and storage of events;
Web Interface: Web-based display interface
It is because of the above components in the LogStash architecture can be independently deployed, it provides a better cluster scalability.
In most cases, LogStash hosts fall into two broad categories:
Agent host (agent host): as the event transporter (shipper), the various log data sent to the central host; just run the Logstash agent (agent) program;
Central host: Runs various components, including intermediate brokers, indexers, search and storage, and Web interfaces, to enable the logging of data Of the receiving, processing and storage.
deploy
Basic environment
Yum install java-1.7.0-openjdk
Java -version # Ensure java version 1.7
Deploy LogStash
# Download
Wget https://download.elasticsearch.org/logstash/logstash/logstash-1.3.1-flatjar.jar -Ologstash.jar
# start up
Java - jar logstash.jar agent - v - f shipper.conf # start shipper
Java -jar logstash.jar agent -v-f indexer.conf # Start indexer
Deploy Redis
# Installation
Yum install redis-server
# start up
/etc/init.d/redis-server start
# Test
$ Redis-cli -h 192.168.12.24
Redis 192.168.12.24:6379> PING
PONG
Deploy Elasticsearch
# Download
Wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.8.noarch.rpm
# Installation
Rpm -ivh elasticsearch-0.90.8.noarch.rpm
# start up
/etc/init.d/elasticsearch status
Start Kibana
# Installation
Java -jar logstash.jar web # LogStash 1.3.1 comes with Kibana
# View
Http://192.168.12.24:9292
Logstash configuration files and plug-ins
Input {
Stdin {}
File {
Type => "syslog"
Path => [ "/ var / log / secure", "/ var / log / messages"]
Exclude => [ "* .gz", "shipper.log"]
}}
Zeromq {
Address => [ "tcp: //192.168.8.145: 8889"]
Mode => "client"
Type => "zmq-input"
Topic => "weblog"
Topology => "pubsub"
Codec => "json"
}}
}}
Filter {
Mutate {
Gsub => [ "message", "APPS weblog", ""]
Gsub => [ "message", "{", ""]
Gsub => [ "message", "}", ""]
}}
}}
Output {
Stdout {debug => true debug_format => "json"}
Elasticsearch {
Cluster => "logstash"
Codec => "json"
}}
}}
Log categories and processing methods
Apache log: custom apache output log format, json output, no filter to participate
Postfix log: can not be customized, need to use filters such as grok filter
Tomcat log: the need to merge multiple logs into an event, and exclude blank lines
Cluster extension
Extended architecture
Precautions
Redis: deploy multiple, only available high-availability role, no load sharing role, you can use ZeroMQ instead
ElasticSearch:
# Check the node status:
Curl -XGET 'http://127.0.0.1:9200/_cluster/health?pretty=true'
Green status: All shards are allocated and are operating normally
Yellow status: Only the primary shard is allocated, such as when the cluster is replicating data between nodes
Red status: There is a shard not allocated
# Cluster monitoring:
Paramedic Tools:
Install: / usr / share / elasticsearch / bin / plugin -install karmi / elasticsearch-paramedic
See: http://log.linuxidc.net: 9200 / _plugin / paramedic / index.html
Bigdesk Tools:
Install: / usr / share / elasticsearch / bin / plugin -install lukas-vlcek / bigdesk
See: http://log.linuxidc.net: 9200 / _plugin / bigdesk / index.html
# Data Retention Policy:
1.LogStash default to create an index for each day, you can manually delete the index
Curl -XDELETE http://127.0.0.1:9200/logstash-2013.12.19
Shell optimization script: https://github.com/cnf/logstash-tools/blob/master/elasticsearch/clean-elasticsearch.sh
2. Optimization index:
Curl -XPOST 'http://127.0.0.1:9200/logstash-2013.12.19/_optimize'
Curl -XPOST 'http://127.0.0.1:9200/_optimize' # optimize all the index
Curl 'http://127.0.0.1:9200/logstash-2013.12.19/_stats?clear=true&store=true&pretty=true' # Check the index size, index too much will affect the optimization time-consuming
3. Default index data directory: / var / lib / elasticsearch / logstash
References
LogStash official website: http: //www.logstash.net/
Elasticsearch official website: http: //www.elasticsearch.org/
Kibana Query Syntax: http: //lucene.apache.org/core/3_6_1/queryparsersyntax.html |
|
|
|