Audit Logs

Much of the security material we've discussed so far helps to protect your data, your system, and its resources when particular events occur. Authentication helps identify and verify clients as connections are opened. Authorization helps secure access to one or more resources with each request. But if something bad were to happen, let's say a rogue client spawns multiple topics and produces fake messages, where someone repeatedly tries to open many connections in quick succession using invalid credentials in an attempt to exhaust broker resources. How would you know your system has been targeted, who was involved, and the sequence of events that has led to your current unfortunate state. Going further, is there any way of being able to anticipate or prevent future attacks as new threats emerge? This is where audit logs can help. An audit log is a record of everything that has happened on your system. With an audit log, you can improve your security posture by first gaining insight. Logs provide insight into what is happening in your system. Just rolled out a new access control list for a new group? Using audit logs, you can go back and see if users have been able to authenticate and if they're authorized to the correct resources. Number two: Improve security. By monitoring and validating operations, you can detect unauthorized operations in the historical record and trace any other interactions associated with the client or host attempting unauthorized operations. As you look at the logs, are there things that stick out or seem strange? With logs, you can identify anomalies and take action as quickly as possible. Number three: understand impact. You can use the logs to debug client interactions and spot unusual activities and system metrics. The logs can help you see who and what is impacted if there are issues. If there's an ongoing issue, the log allows you to see who and what services are impacted, what got you to that point, and then allow you to communicate with stakeholders as the situation progresses. Number four: prove compliance. Logs can help you generate audit reports in line with internal policies and external regulations and create an official record if there is a security breach. Apache Kafka doesn't provide anything called an audit log directly out of the box. It does, however, support comprehensive application logging using Log4J. Log4J can be configured with multiple appenders that deliver log events from different logger instances to different destinations, typically files. There are two logger instances, Kafka.authorizer.logger, which is used for authorization logging and Kafka.request.logger. By configuring separate appenders for the Kafka authorized logger and Kafka request logger, logger instances, in the log.4J properties file on each broker, you can capture detailed authorization and request information for auditing, monitoring, and debugging purposes. The authorization logger generates info entries for operations that are denied and debug entries for operations which access was granted. Both types of entry include details of the user principle, the client host, the operation being attempted and the resource, such as topic, to which the operation was directed. The request logger adds details of the user principle and the client host at the debug level and full details of the request when logging at the trace level. Kafka's application logging provides a solid basis for creating and analyzing analogs to meet your specific audit needs, whether it's anomaly detection, debugging, or compliance. Of course there are a few things to be aware of. If you're logging to files on each broker, you should ensure there's sufficient disc space and an appropriate Log4J retention policy so that you don't fill the discs. The request logger in particular can generate a lot of data very quickly, so you may want to consider enabling it only for debugging purposes or using a very small retention window. Kafka's application logs are per broker. To get a complete picture of what is happening in your system, you'll need to consolidate the logs from all brokers in a cluster to help analyze and visualize this consolidated output. You could even direct the log output to a Kafka topic or another cluster. We recommend using a commercial or open source log analysis and observability framework, such as the ELK stack, that is Elasticsearch, Logstash, and Kibana to help analyze and visualize this consolidated output. You could even direct the log output to a Kafka topic in another cluster and then perform streaming analysis on your audit events.