Skip to main content

Ingest API

EndpointUse it forBody
POST /api/v1/ingestStructured eventsNDJSON, one JSON object per line
POST /api/v1/ingest/rawLog files and streamsLog text, one event per line
POST /api/v1/ingest/hecSplunk HEC sendersOne HEC event object per line

Shippers that speak Elasticsearch, Splunk HEC or OTLP can also use the compatibility endpoints. When authentication is on, the ingest endpoints need a key with ingest scope.

How events are stored​

Every index keeps its events in a columnar store of its own. An index comes into being with its first events, taking the server's storage.sort_key, storage.time_fields and storage.partition as its settings, or with PUT /indexes/{index}, which gives it settings of its own. The settings are written to the index's directory before it takes events, and the index keeps them: changing the storage settings affects only indexes created afterwards.

  • Time. An event's time is the value of the first of the index's time fields that it holds; storage.time_fields gives _time, timestamp, @timestamp, time and ts. An event without one gets the time it arrived.
  • Text. A line that is not a JSON object is kept as an event holding the text in _raw. A JSON event has no _raw.
  • Fields. The members of a nested object are read as columns named by their path, such as order.id. A query result holds only the fields an event has: a field the event lacks is absent, not null.
  • Retention. Events whose own time is older than retention (7 days by default) are removed, partition by partition, within about a minute of arriving or when the server starts. Each removal is logged at WARN with the index, the event count and the cutoff. retention: 0s keeps everything.
  • Visibility. Ingested events are written within storage.flush_interval (one second by default). The server writes the pending events before it runs a query, so a query sees every event whose ingest request has been answered. The Elasticsearch bulk endpoints, and Splunk HEC requests that carry an acknowledgement channel, make the events durable before they answer.

Index names hold letters, digits, ., - and _, do not start with ., and are at most 255 bytes. Any other name answers 400 VALIDATION_ERROR.

POST /ingest​

Takes NDJSON and appends it, line by line and unchanged, to one index: the index query parameter, else the X-Index header, else main.

curl -s -X POST 'localhost:3100/api/v1/ingest?index=demo' --data-binary $'{"_time":"2026-11-17T08:00:00Z","level":"info","msg":"user login","user_id":42}
{"timestamp":"2026-11-17T08:00:05Z","level":"error","msg":"payment failed","order":{"id":7,"total":19.5}}
plain text line
'
{"data": {"accepted": 3, "failed": 0}}
curl -s localhost:3100/api/v1/query \
-H 'Content-Type: application/json' \
-d '{"q": "source=demo | sort _time"}'
{
"data": {
"type": "events",
"events": [
{"_time": "2026-11-17T08:00:00Z", "level": "info", "msg": "user login", "user_id": 42},
{"_time": "2026-11-17T08:00:05Z", "level": "error", "msg": "payment failed", "order.id": 7, "order.total": 19.5, "timestamp": "2026-11-17T08:00:05Z"},
{"_time": "2026-11-17T15:46:35.786981Z", "_raw": "plain text line"}
],
"total": 3,
"has_more": false
},
"meta": {"took_ms": 0.103, "query_id": "qry_f12636a8a2a0b0b4", "rows": 3}
}

accepted counts the lines that are not blank, less those the store rejected; failed counts the rejected ones. Blank lines are skipped.

POST /ingest and POST /ingest/raw are bounded by ingest.max_body_size alone, as sent and after gzip, zstd or snappy decoding. A body over it stores nothing and answers 413, naming the setting:

{"error": {"type": "request_body_too_large", "reason": "compressed body exceeds limit 4096 set by ingest.max_body_size"}, "status": 413}

A body sent without Content-Length, and a compressed body, is appended as it is read. When it grows past the limit the reply is 413 INVALID_REQUEST; when it is cut short or does not decode the reply is 400 INVALID_REQUEST. Either message ends with the lines taken before that point:

{"error": {"code": "INVALID_REQUEST", "message": "request body is cut short or does not decode; the 17371 lines before that were taken"}}

Sending such a body again repeats the lines already taken.

POST /ingest/raw​

Takes log text, one event per line. The index is the X-Index header, else the index query parameter, else main.

HeaderDefaultDescription
X-IndexmainTarget index
X-Sourcehttpsource of the text lines
X-Source-Typerawsourcetype of the text lines
X-Hosthost of the text lines

Each header holds at most 256 characters; a longer one answers 400 VALIDATION_ERROR.

  • A line that is a JSON object (its first non-blank character is {) passes unchanged, without the labels.
  • Any other line becomes an event with the line in _raw, the labels as source, sourcetype and host, and:
    • its timestamp as _time, found at the start of the line or after a space or a double quote within its first 100 characters: RFC 3339 with millisecond or finer precision or a numeric offset (2026-11-17T08:00:00.123Z, 2026-11-17T08:00:00+01:00), 2026-11-17 08:00:00, or the syslog form Nov 17 08:00:00. A line without one gets the time it arrived;
    • its key=value pairs as fields. Integers, decimals and true/false keep their type, so status>=500 compares numbers.
  • Blank lines are skipped. A line longer than 1 MiB ends the request.
curl -s -X POST localhost:3100/api/v1/ingest/raw \
-H 'X-Index: rawdemo' \
-H 'X-Source: api-gateway' \
-H 'X-Host: gw-01' \
--data-binary $'2026-11-17T08:00:00.000Z level=warn path=/api/users latency_ms=812 slow request\n{"level":"info","msg":"json passes unchanged"}\n'
{"data": {"accepted": 2, "failed": 0}}

The two events read back as:

{"_raw": "2026-11-17T08:00:00.000Z level=warn path=/api/users latency_ms=812 slow request", "_time": "2026-11-17T08:00:00Z", "host": "gw-01", "latency_ms": 812, "level": "warn", "path": "/api/users", "source": "api-gateway", "sourcetype": "raw"}
{"_time": "2026-11-17T15:46:41.906133Z", "level": "info", "msg": "json passes unchanged"}

Pipe command output the same way:

kubectl logs deploy/api --since=1h | \
curl -s -X POST localhost:3100/api/v1/ingest/raw \
-H 'X-Source: api-gateway' \
--data-binary @-

When the body is cut off, by ingest.max_body_size or by a line over 1 MiB, after some events were stored, the response is 200 with truncated: true and a warning. When nothing was stored, it answers 400 INVALID_REQUEST.

POST /ingest/hec​

Splunk HEC event ingest. Each line of the body is one HEC event object; lines that are not valid JSON are skipped and counted in skipped_lines.

curl -s -X POST localhost:3100/api/v1/ingest/hec \
-d '{"event": "via api", "host": "web-02"}'
{"code": 0, "text": "Success"}

This path authenticates like every other /api/v1 endpoint and does not read Authorization: Splunk tokens. It accepts events whatever ingest.splunk_hec.enabled says. Shippers configured for Splunk usually point at /services/collector; see Splunk HEC for those routes, acknowledgements and the mapping of HEC fields to events.

GET /indexes​

Lists the indexes with what they hold:

curl -s localhost:3100/api/v1/indexes
{
"data": {
"indexes": [
{"name": "backend", "events": 26, "bytes": 3978, "parts": 1, "merges_running": 0, "runs_to_merge": 0, "first_event": "2026-02-10T08:12:03.045Z", "last_event": "2026-02-10T10:45:00.5Z", "sort_key": [], "time_fields": ["_time", "timestamp", "@timestamp", "time", "ts"], "partition": "1d"},
{"name": "main", "events": 1000, "bytes": 46638, "parts": 2, "merges_running": 0, "runs_to_merge": 2, "first_event": "2025-01-15T08:00:00.105Z", "last_event": "2026-11-17T15:43:40.839562Z", "sort_key": [], "time_fields": ["_time", "timestamp", "@timestamp", "time", "ts"], "partition": "1d"}
]
}
}
FieldDescription
nameIndex name
eventsEvents stored
bytesSize of the index's files
partsColumnar files the index holds
merges_runningMerges of the index's files under way
runs_to_mergeSorted runs that background merging is still to merge, those of a past partition that waits for events to stop arriving included; the index is quiet when this and merges_running are both 0
first_event, last_eventTimes of the earliest and latest event; absent for an index without stored events
sort_key, time_fields, partitionThe settings of the index, see PUT /indexes/{index}

Query an index with source=<name>:

curl -s localhost:3100/api/v1/query \
-H 'Content-Type: application/json' \
-d '{"q": "source=backend | stats count() by level"}'

GET /indexes/{index}​

Describes one index as an entry of GET /indexes does:

curl -s localhost:3100/api/v1/indexes/bluesky
{"data": {"name": "bluesky", "events": 2, "bytes": 889, "filter_bytes": 32, "parts": 1, "merges_running": 0, "runs_to_merge": 0, "first_event": "2024-11-21T16:25:49.000167Z", "last_event": "2024-11-21T16:25:49.00028Z", "sort_key": ["kind", "commit.operation", "commit.collection", "did"], "time_fields": ["time_us"], "partition": "1d"}}

An index that does not exist answers 404 NOT_FOUND.

PUT /indexes/{index}​

Creates an index with settings of its own, before any event arrives. Declare as the sort key the fields that the queries of the index filter and group by, the most selective last, so that its files are written in that order:

curl -s -X PUT localhost:3100/api/v1/indexes/bluesky \
-H 'Content-Type: application/json' \
-d '{"sort_key": ["kind", "commit.operation", "commit.collection", "did"], "time_fields": ["time_us"], "partition": "1d"}'
{"data": {"name": "bluesky", "events": 0, "bytes": 0, "filter_bytes": 0, "parts": 0, "merges_running": 0, "runs_to_merge": 0, "sort_key": ["kind", "commit.operation", "commit.collection", "did"], "time_fields": ["time_us"], "partition": "1d"}}
SettingDefaultValue
sort_keystorage.sort_keyUp to 32 field paths whose values order the rows of the index, the first the most significant; [] orders them by time
time_fieldsstorage.time_fieldsUp to 32 field paths whose value becomes the time of an event, tried in their order
partitionstorage.partitionThe span of time whose events are kept apart, from 1m to 366d, such as 1h or 1d; retention drops whole partitions

Every setting is optional, and the body may be left out. A field path names a nested member by its path, such as commit.collection; it has no empty segment and no control character, and a list names it once.

The call answers 201 when it creates the index. The settings of an index do not change: on an index that exists, the call answers 200 with the index when the index has every setting the body gives, and 409 CONFLICT, naming the first setting that differs, otherwise:

{"error": {"code": "CONFLICT", "message": "columnar: the index exists with other settings: index bluesky has sort_key [kind commit.operation commit.collection did], not [did]", "suggestion": "The settings of an index do not change; create another index for another design."}}

It needs ingest or admin scope when authentication is on.

POST /indexes/{index}/compact​

Writes the events ingested so far, merges the files of every partition of the index into one sorted run, and answers when that is done. Background merging gets there too, but only after a partition has gone without new events for a while; call this at the end of a bulk load so that the queries that follow read merged files. The request takes as long as the merge; queries that run meanwhile read the files as they were until the merge commits. It needs ingest or admin scope when authentication is on.

curl -s -X POST localhost:3100/api/v1/indexes/main/compact
{"data": {"index": "main", "parts_before": 6, "parts": 1, "bytes_before": 48210, "bytes": 46638, "seconds": 0.08}}
FieldDescription
parts_before, bytes_beforeFiles and their size once the pending events were written
parts, bytesFiles and their size after the merge
secondsTime the request took, the write of the pending events included

An index that does not exist answers 404 NOT_FOUND; a server that is shutting down answers 503 SHUTTING_DOWN and leaves the files as they were.

Errors​

HTTPCodeWhen
400VALIDATION_ERRORAn invalid index name or index setting, or an X-Index, X-Source, X-Source-Type or X-Host header longer than 256 characters
400INVALID_JSONThe body of PUT /indexes/{index} is not a JSON object of the settings
400INVALID_REQUESTThe body could not be read
400The Content-Encoding is not gzip, zstd or snappy: {"error": {"type": "invalid_request", ...}, "status": 400}
401AUTH_REQUIRED, INVALID_TOKENMissing or wrong credentials
403FORBIDDENThe key lacks ingest scope
404NOT_FOUNDGET /indexes/{index} or POST /indexes/{index}/compact names an index that does not exist
409CONFLICTPUT /indexes/{index} asks for other settings than the index has
413INVALID_REQUESTThe body of POST /ingest, sent without Content-Length, grows past ingest.max_body_size
413The body is over its size limit: {"error": {"type": "request_body_too_large", ...}, "status": 413}, with a reason that names the setting
429TOO_MANY_REQUESTSThe per-IP rate limit (http.rate_limit) was hit
503SHUTTING_DOWNThe server is stopping; retry after Retry-After
500INTERNAL_ERRORThe events could not be stored