Data governance in a crisis: when every second counts

Imagine payment failures across Australia at 2 PM. The on-call team finds a location field full of random characters, and the team that deployed the change five hours earlier in Europe has gone home. Governance tools let them work it out. A data catalog or schema registry shows the field is a string classified as personally identifiable information (PII), which suggests it's encrypted. Just four points of location and time data are enough to uniquely identify a person 95% of the time, so location counts as PII.

The on-call team doesn't have the encryption keys, which is the principle of least privilege and role-based access control (RBAC) at work. Data lineage shows that the messages came from a newly deployed regional fraud detection service. The data steward listed in the schema's metadata explains that the service selectively encrypts the location field to comply with new (for this example, made-up) Australian regulations. The root cause: the decryption logic reads its regions from a configuration file, and Australia was missing.

This scenario demonstrates the three critical dimensions of streaming data governance: the right people, the right data, and the right time. The "right people" dimension involves protecting PII through encryption, implementing role-based access control following the principle of least privilege, and establishing clear data stewardship. The "right data" dimension ensures quality through proper validation and discovery through catalogs and metadata. But the "right time" dimension is what transforms governance from a theoretical exercise into operational necessity. Traditional governance tools built for batch processing often update on daily schedules, meaning data catalogs and lineage information could be 24 hours out of date. During a production incident, that delay could mean investigating the wrong service, missing key schema changes, or contacting the wrong team. This team had timely information and averted the crisis, but remove any one of the three dimensions and the system begins to crumble.

Topics:

• Production incident walkthrough: payment failures in Australia
• Data catalogs and schema registries for self-service discovery
• PII, least privilege, and role-based access control (RBAC)
• Data lineage and data stewardship
• Why governance tools need real-time information
• The three dimensions: right people, right data, right time

Resources

• Unique in the Crowd: The privacy bounds of human mobility - Research showing four spatio-temporal points are enough to uniquely identify 95% of individuals
• IBM Cost of a Data Breach Report - IBM's annual study of what data breaches cost organizations; the 2024 edition put the global average at $4.88 million
• Verizon Data Breach Investigations Report - Annual analysis of real-world breaches, how they happen, and who they affect
• Governing Data Streams - Confluent Developer course on schema registries, data catalogs, and data lineage for data streams
• Client-Side Field Level Encryption (CSFLE) - Confluent Developer course on encrypting individual fields in messages so only users with the keys can read them