Get Started Free

Event Joiner

Event Streams may need to be joined (i.e. enriched) with a Table or another Event Stream in order to provide more comprehensive details about their Events.

Problem

How can I enrich an Event Stream or Table with additional context?

Solution

event joiner

We can combine Events in a stream with a Table or another Event Stream by performing a join between the two. The join is based on a key shared by the original Event Stream and the other Event Stream or Table. We can also provide a window buffering mechanism based on timestamps, so that we can produce join results when Events from both Event Streams aren't immediately available. Another approach is to join an Event Stream and a Table that contains more static data, resulting in an enriched Event Stream.

Implementation

With Apache Flink® SQL, we can create a continuously updating Stream of Events from an existing Kafka topic (in this example, note the similarity to fact tables in data warehouses):

CREATE TABLE ratings (
    movie_id INT NOT NULL,
    rating FLOAT
);

We can then create a Table from another existing Kafka topic that changes less frequently. This Table serves as our reference data (similar to dimension tables in data warehouses).

CREATE TABLE movies (
    movie_id INT NOT NULL,
    title STRING,
    release_year INT  
);

To create a Stream of enriched Events, we perform a join between the Event Stream and the Table.

SELECT ratings.movie_id as id, title, rating
FROM ratings
LEFT JOIN movies ON ratings.movie_id = movies.movie_id;

Considerations

  • We can perform an inner or left-outer join between an Event Stream and a Table.
  • Joins are also useful for initiating subsequent processing when two or more corresponding Events arrive on different Event Streams or Tables.

References

Confluent Cloud is a fully managed Apache Kafka service available on all three major clouds. Try it for free today.

Try it for free