Analyze Kubernetes Audit Logs
The Kubernetes API server's log audit backend writes one JSON event per line.
When the backend writes to its common /var/log/kubernetes/audit/audit.log
path, LynxDB can analyze the file in pipe mode, without starting a server or
ingesting the data first.
See the Kubernetes documentation for configuring the audit log
backend
and the audit Event field
reference.
Kubernetes records an anonymous
request
with the username system:anonymous and membership in the
system:unauthenticated group.
Each audit event is stored as the JSON object it is. Top-level keys such as
auditID, verb, userAgent and requestURI are fields, the members of the
nested user, objectRef and responseStatus objects are fields named by
their path, such as user.username and objectRef.resource, and
sourceIPs is an array.
Find anonymous API requests
Count anonymous requests per user agent and source IP:
sudo cat /var/log/kubernetes/audit/audit.log | lynxdb query '
source=main
| where user.username = "system:anonymous"
| expand sourceIPs
| stats dc(auditID) as requests by userAgent, sourceIPs
| sort - requests, sourceIPs'
requests,sourceIPs,userAgent
2,203.0.113.7,curl/8.4.0
1,10.0.0.4,curl/8.4.0
1,198.51.100.23,python-requests/2.31
The pipeline does the following:
wherekeeps the anonymous requests.expand sourceIPsemits one row per address in thesourceIPsarray.dc(auditID)counts distinct API requests for each user agent and address. A request that lists several addresses appears under each of them.
To see which resource types anonymous clients asked for:
sudo cat /var/log/kubernetes/audit/audit.log | lynxdb query '
source=main
| where user.username = "system:anonymous"
| stats count() as requests by userAgent, objectRef.resource
| sort userAgent, objectRef.resource'
objectRef.resource,requests,userAgent
,1,curl/8.4.0
secrets,1,curl/8.4.0
pods,1,python-requests/2.31
Requests to non-resource URLs such as /version have no objectRef, so
they form the group whose objectRef.resource is empty.
Kubernetes documents auditID as a unique ID for each request. It also warns
that proxy entries in sourceIPs can be supplied by the client and that
userAgent is untrusted client input. Treat both fields as investigation
clues, not identity proof.
Summarize failed requests
To see which API operations and resources produce error responses, group by the response code:
sudo cat /var/log/kubernetes/audit/audit.log | lynxdb query '
source=main
| where responseStatus.code >= 400
| stats count() as requests by responseStatus.code, verb, objectRef.resource
| sort - requests, objectRef.resource'
objectRef.resource,requests,responseStatus.code,verb
deployments,1,409,patch
pods,1,403,list
secrets,1,403,list
Keep audit logs on a server
To keep audit events, ship the file to a server, for example with
lynxdb ingest --index k8s-audit, and query source=k8s-audit. Audit events
carry their time in requestReceivedTimestamp, which is not one of the
default time fields; add it to storage.time_fields so that time ranges
select events by it:
storage:
time_fields: [requestReceivedTimestamp, _time, timestamp, "@timestamp", time, ts]
See Working with JSON Logs for more on nested fields,
arrays and expand.