Filebeat to LynxDB
If you already run Filebeat
Change the Elasticsearch host to LynxDB and keep the output type
elasticsearch.
output.elasticsearch:
- hosts: ["http://elasticsearch:9200"]
+ hosts: ["http://lynxdb:3100"]
+ allow_older_versions: true
Restart Filebeat, then check that events arrive:
lynxdb indexes
lynxdb query 'source=filebeat-* | stats count() as total by log.file.path'
Full annotated config
# Filebeat configuration for LynxDB.
# Change only hosts if LynxDB is not reachable at this address.
filebeat.inputs:
- type: filestream
id: lynxdb-files
paths:
- /var/log/*.log
output.elasticsearch:
hosts: ["http://lynxdb:3100"]
allow_older_versions: true
# LynxDB accepts the setup probes, but there are no templates or ILM
# policies to install.
setup.template.enabled: false
setup.ilm.enabled: false
Where the events go
LynxDB accepts POST /_bulk, POST /<index>/_bulk and the setup requests
Filebeat sends at startup. Each document is stored in the LynxDB index named
by its bulk _index, which for Filebeat is usually filebeat-<version>.
lynxdb indexes lists the exact names; source=filebeat-* reads them all.
Each document is stored as sent. Its @timestamp sets the event time, its
message holds the log line, and the members of nested objects are fields
named by their path, such as log.file.path, host.name and agent.version
(see Working with JSON Logs):
lynxdb query 'source=filebeat-* | fields _time, host.name, log.file.path, message | head 2'
Troubleshooting
If Filebeat rejects the server version, set allow_older_versions: true or
change the version LynxDB advertises with ingest.es_compat.advertised_version
(default 8.15.0).
If startup logs mention ILM or templates, set setup.ilm.enabled: false and
setup.template.enabled: false. LynxDB answers the common setup endpoints,
but disabling setup avoids extra requests.
Supported behavior
Supported: Elasticsearch bulk ingest, gzip, zstd and snappy request bodies,
Filebeat's setup probes, and _id values echoed in bulk responses.
Not supported: the Elasticsearch query DSL, ILM state, template storage,
_id-based deduplication, document updates and deletes.