2019-09-10 06:57:26 +00:00
|
|
|
# Ansible role: Logstash
|
|
|
|
|
2024-05-08 08:33:38 +00:00
|
|
|
[![Version](https://img.shields.io/badge/latest_version-1.1.0-green.svg)](https://code.waks.be/nishiki/ansible-role-logstash/releases)
|
|
|
|
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://code.waks.be/nishiki/ansible-role-logstash/src/branch/main/LICENSE)
|
2024-05-09 09:47:57 +00:00
|
|
|
[![Build](https://code.waks.be/nishiki/ansible-role-logstash/actions/workflows/molecule.yml/badge.svg?branch=main)](https://code.waks.be/nishiki/ansible-role-logstash/actions?workflow=molecule.yml)
|
2019-09-10 06:57:26 +00:00
|
|
|
|
|
|
|
Install and configure logstash
|
|
|
|
|
|
|
|
## Requirements
|
|
|
|
|
2024-05-08 08:33:38 +00:00
|
|
|
- Ansible >= 2.9
|
|
|
|
- Debian
|
|
|
|
- Bookworm
|
2019-09-10 06:57:26 +00:00
|
|
|
|
|
|
|
## Role variables
|
|
|
|
|
2024-05-08 08:33:38 +00:00
|
|
|
- `logstash_major_version` set major version to install- (default: `7`)
|
|
|
|
- `logstash_plugins` - array with the plugins to install
|
2019-09-10 06:57:26 +00:00
|
|
|
|
|
|
|
```
|
|
|
|
- name: logstash-output-influxdb
|
|
|
|
state: present
|
|
|
|
```
|
|
|
|
|
2024-05-08 08:33:38 +00:00
|
|
|
- `logstash_config` - hash with the configuration (see [logstash documentation](https://www.elastic.co/guide/en/logstash/current/configuration.html))
|
2019-09-10 06:57:26 +00:00
|
|
|
|
|
|
|
```
|
|
|
|
path.data: /var/lib/logstash
|
|
|
|
path.logs: /var/log/logstash
|
|
|
|
```
|
|
|
|
|
|
|
|
- `logstash_inputs` - hash with the inputs configurations
|
|
|
|
|
|
|
|
```
|
|
|
|
file:
|
|
|
|
path: /var/log/syslog
|
|
|
|
beats:
|
|
|
|
port: 5444
|
|
|
|
```
|
|
|
|
|
|
|
|
- `logstash_outputs` - array with the outputs configuration
|
|
|
|
|
|
|
|
```
|
|
|
|
- >
|
|
|
|
file {
|
|
|
|
path => "/var/log/syslog"
|
|
|
|
codec => "json"
|
|
|
|
}
|
|
|
|
```
|
|
|
|
|
|
|
|
- `logstash_filters` - array with the filters configuration
|
|
|
|
|
|
|
|
```
|
|
|
|
- >
|
|
|
|
grok {
|
|
|
|
match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }
|
|
|
|
}
|
|
|
|
```
|
|
|
|
|
2024-02-15 13:40:18 +00:00
|
|
|
- `logstash_pattern` - hash with grok patterns
|
2021-08-24 14:55:01 +00:00
|
|
|
|
|
|
|
```
|
|
|
|
postfix: |
|
|
|
|
# Syslog stuff
|
|
|
|
PROCESS ([\w._\/%-]+)
|
|
|
|
COMPID postfix\/%{PROCESS:process}(?:\[%{NUMBER:pid}\])?
|
|
|
|
POSTFIX (?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{COMPID}:
|
|
|
|
```
|
|
|
|
|
2024-02-15 13:40:18 +00:00
|
|
|
- `logstash_logging_config` - hash with logging config (log4j2)
|
|
|
|
|
|
|
|
```yaml
|
2024-05-08 08:33:38 +00:00
|
|
|
status: "error"
|
|
|
|
name: "LogstashPropertiesConfig"
|
|
|
|
appender.rolling.type: "RollingFile"
|
|
|
|
appender.rolling.name: "plain_rolling"
|
|
|
|
appender.rolling.fileName: "${sys:ls.logs}/logstash-plain.log"
|
|
|
|
appender.rolling.filePattern: "${sys:ls.logs}/logstash-plain-%d{yyyy-MM-dd}-%i.log.gz"
|
|
|
|
appender.rolling.policies.type: "Policies"
|
|
|
|
appender.rolling.policies.time.type: "TimeBasedTriggeringPolicy"
|
|
|
|
appender.rolling.policies.time.interval: "1"
|
|
|
|
appender.rolling.policies.time.modulate: true
|
|
|
|
appender.rolling.layout.type: "PatternLayout"
|
|
|
|
appender.rolling.layout.pattern: "[%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]}%notEmpty{[%X{plugin.id}]} %m%n"
|
|
|
|
appender.rolling.policies.size.type: "SizeBasedTriggeringPolicy"
|
|
|
|
appender.rolling.policies.size.size: "100MB"
|
|
|
|
appender.rolling.strategy.type: "DefaultRolloverStrategy"
|
|
|
|
appender.rolling.strategy.max: 30
|
|
|
|
appender.rolling.avoid_pipelined_filter.type: "PipelineRoutingFilter"
|
|
|
|
appender.routing.type: "PipelineRouting"
|
|
|
|
appender.routing.name: "pipeline_routing_appender"
|
|
|
|
appender.routing.pipeline.type: "RollingFile"
|
|
|
|
appender.routing.pipeline.name: "appender-${ctx:pipeline.id}"
|
|
|
|
appender.routing.pipeline.fileName: "${sys:ls.logs}/pipeline_${ctx:pipeline.id}.log"
|
|
|
|
appender.routing.pipeline.filePattern: "${sys:ls.logs}/pipeline_${ctx:pipeline.id}.%i.log.gz"
|
|
|
|
appender.routing.pipeline.layout.type: "PatternLayout"
|
|
|
|
appender.routing.pipeline.layout.pattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
|
|
|
|
appender.routing.pipeline.policy.type: "SizeBasedTriggeringPolicy"
|
|
|
|
appender.routing.pipeline.policy.size: "100MB"
|
|
|
|
appender.routing.pipeline.strategy.type: "DefaultRolloverStrategy"
|
|
|
|
appender.routing.pipeline.strategy.max: 30
|
|
|
|
rootLogger.level: "${sys:ls.log.level}"
|
|
|
|
rootLogger.appenderRef.rolling.ref: "${sys:ls.log.format}_rolling"
|
|
|
|
rootLogger.appenderRef.routing.ref: "pipeline_routing_appender"
|
2024-02-15 13:40:18 +00:00
|
|
|
```
|
|
|
|
|
2019-09-10 06:57:26 +00:00
|
|
|
## How to use
|
|
|
|
|
|
|
|
```
|
|
|
|
- hosts: server
|
|
|
|
roles:
|
|
|
|
- logstash
|
|
|
|
```
|
|
|
|
|
|
|
|
## Development
|
|
|
|
|
2021-08-24 14:58:34 +00:00
|
|
|
### Test with molecule and docker
|
2019-09-10 06:57:26 +00:00
|
|
|
|
2024-05-08 08:33:38 +00:00
|
|
|
- install [docker](https://docs.docker.com/engine/installation/)
|
|
|
|
- install `python3` and `python3-pip`
|
|
|
|
- install molecule and dependencies `pip3 install molecule molecule-docker docker ansible-lint pytest-testinfra yamllint`
|
|
|
|
- run `molecule test`
|
2019-09-10 06:57:26 +00:00
|
|
|
|
|
|
|
## License
|
|
|
|
|
|
|
|
```
|
|
|
|
Copyright (c) 2019 Adrien Waksberg
|
|
|
|
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License");
|
|
|
|
you may not use this file except in compliance with the License.
|
|
|
|
You may obtain a copy of the License at
|
|
|
|
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
See the License for the specific language governing permissions and
|
|
|
|
limitations under the License.
|
|
|
|
```
|