Apply Lambda Functions to Arrays and Maps
Allison Walther
Integration Architect (Presenter)
Apply Lambda Functions to Arrays and Maps
You've learned that ksqlDB can apply filters to each event in a stream based on values in the events—even filtering events in nested fields. But what about events in an ARRAY, or a MAP field? You can use lambda functions for this. Lambda functions allow you to use filters, transformations, and reductions.
Transform Function
A transform will change the key or value of every element in a MAP or ARRAY:
CREATE STREAM transformed
AS SELECT id,
TRANSFORM(map_field,(k,v)=> UCASE(k), (k, v)=> (v * 2))
FROM stream1 EMIT CHANGES;Note that for a MAP field, two lambda expressions are required: one for the key and one for the value. If no change is required for the key or the value, the lambda can just return the existing data.
Reduce Function
Reduce will take all of the elements from an ARRAY or MAP and produce a single result, based on the lambda expression:
CREATE STREAM reduced
AS SELECT name,
REDUCE(scores,0,(s,x)=> (s+x)) AS total
FROM stream2 EMIT CHANGES;For an ARRAY, the lambda takes two arguments: the ARRAY element and an accumulator. For a MAP, the lambda takes three arguments: the key, the value, and an accumulator.
Filter Function
The filter function takes an ARRAY or MAP and returns the same type, but filled with only the elements that meet the filter's criteria, which are given in the lambda expression.
CREATE STREAM filtered
AS SELECT id,
FILTER(numbers,x => (x%2 = 0)) AS even_numbers
FROM stream3 EMIT CHANGES;Use the promo code KSQLDB101 & CONFLUENTDEV1 to get $25 of free Confluent Cloud usage and skip credit card entry.
Be the first to get updates and new content
We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.
Apply Lambda Functions to Arrays and Maps
Hi, I'm Allison Walther with Confluent. Let's talk about ksqlDB and Lambdas. We already know that ksqlDB can perform transformations and filters on each event in a stream based on values and the events, even values and nested fields. But what about elements of an array or map field? That's where ksqlDB Lambdas come in, we can perform transformations, filters, or reductions on every element in an array or map field in our events, the transform function will change the key and our value of each element in a map or the value of each element in an array field. Based on the Lambda expression given. Note for a map field, two Lambda expressions are required. One for the key and one for the value, if no changes desired for either key or value, the Lambda can just return the existing data, for reduce we'll take all of the elements of an array or map and produce a single result based on the Lambda expression, for an array, the Lambda will take two arguments, the array element, and an accumulator. For a map, the Lambda will take three arguments, the key, value and an accumulator. The filter function we'll take an array or map and return the same type, but with only elements that meet the filter criteria given in the Lambda expression, ksqlDB Lambdas, open up a whole new way to work with data in our event streams, let's use transform, reduce and filter in our exercise. That's it for this lesson.