Staff Software Practice Lead
When working with infinite streams of data, some operations require us to split the stream into more manageable chunks. This is done using time windows and it allows us to perform aggregations that would otherwise be impossible. When working with windows, Flink uses something known as a watermark to track time throughout the stream. In this video, you will learn the basic types of windows that can be applied to a stream, as well as how to enable watermarks.
Topics:
stream.windowAll(timeWindow);
stream
.keyBy(record -> record.key)
.window(timeWindow)
stream
.keyBy(record -> record.key)
.window(
TumblingEventTimeWindows
.of(Time.seconds(5))
)
stream
.keyBy(record -> record.key)
.window(
SlidingEventTimeWindows
.of(Time.seconds(10), Time.seconds(5))
)
stream
.keyBy(record -> record.key)
.window(
EventTimeSessionWindows
.withGap(Time.minutes(10))
)
stream1.join(stream2)
.where(elem1 -> elem1.getKey()).equalTo(
elem2 -> elem2.getKey()
)
.window(timeWindow)
.apply(new JoinFunction<Input1, Input2, Output>() {
@Override
public Output join(Input1 input1, Input2 input2) {
...
}
});
WatermarkStrategy<DataType> watermarkStrategy =
WatermarkStrategy.forMonotonousTimestamps()
WatermarkStrategy<DataType> watermarkStrategy =
WatermarkStrategy.forBoundedOutOfOrderness(
Duration.ofSeconds(10)
)
WatermarkStrategy<DataType> watermarkStrategy =
WatermarkStrategy.forBoundedOutOfOrderness(
Duration.ofSeconds(10)
)
.withTimestampAssigner(
(event, timestamp) -> timestamp
)
WatermarkStrategy<DataType> watermarkStrategy =
WatermarkStrategy.forBoundedOutOfOrderness(
Duration.ofSeconds(10)
)
.withTimestampAssigner(
(event, timestamp) -> timestamp
)
.withIdleness(Duration.ofSeconds(10));
We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.