Get Started Free
‹ Back to courses
course: ksqlDB 101

Transforming Data with ksqlDB

2 min
Allison

Allison Walther

Integration Architect (Presenter)

Robin Moffatt

Robin Moffatt

Principal Developer Advocate (Author)

Transforming Data with ksqlDB

ksqlDB allows you to transform events in one stream and then send them to a new stream. For example, you may want to change a Unix timestamp to a human-readable format, or you may want to remove sensitive fields from a stream for broader use within a company.

ksqlDB can be used to perform many common transformations required when handling data, including:

  • Changing data types (CAST)
  • Reformatting date/time fields (TIMESTAMPTOSTRING)
  • Changing field names (AS)
  • Dropping fields
  • Concatenating fields (CONCAT)

When creating a new stream from an existing one, you can pick and choose the fields you’d like to preserve in the new stream. You can also use functions to modify data or to create derived fields.

For example, here is how to convert the timestamps on your orders from the Unix epoch value to a more friendly data format:

orders-no-address

There is really no limit to the transformations you can perform with ksqlDB.

Use the promo code KSQLDB101 to get $25 of free Confluent Cloud usage

Get Started

Get started with Confluent Cloud

Try free

Total 28

  1. Introduction to ksqlDB
    2 min
  2. Interacting with ksqlDB
    2 min
  3. Hands On: Interacting with ksqlDB
    5 min
  4. Creating, Exporting, and Importing Data
    2 min
  5. Filtering with ksqlDB
    1 min
  6. Hands On: Filtering with ksqlDB
    1 min
  7. Lookups and Joins with ksqlDB
    2 min
  8. Hands On: Lookups and Joins with ksqlDB
    2 min
  9. Transforming Data with ksqlDB
    2 min
  10. Hands On: Transforming Data with ksqlDB
    2 min
  11. Flatten Nested Records with ksqlDB
    1 min
  12. Hands On: Flatten Nested Records with ksqlDB
    2 min
  13. Converting Data Formats with ksqlDB
    2 min
  14. Hands On: Converting Data Formats with ksqlDB
    2 min
  15. Merging Two Streams with ksqlDB
    1 min
  16. Hands On: Merging Two Streams with ksqlDB
    2 min
  17. Splitting Two Streams with ksqlDB
    2 min
  18. Hands On: Splitting Two Streams with ksqlDB
    2 min
  19. Streams and Tables
    2 min
  20. Stateful Aggregations (Materialized Views)
    2 min
  21. Hands On: Stateful Aggregations (Materialized Views)
    2 min
  22. Push Queries and Pull Queries
    2 min
  23. Hands On: Push Queries and Pull Queries
    2 min
  24. Under the Covers of ksqlDB
    2 min
  25. ksqlDB Code Lifecycle
    2 min
  26. Apply Lambda Functions to Arrays and Maps
    2 min
  27. Hands On: Apply Lambda Functions to Arrays and Maps
    2 min
  28. Monitoring ksqlDB
    1 min

Related Resources

Presentation

Inside ksqlDB

Tutorial

How to Transform a Stream of Events

Docs

CREATE STREAM AS SELECT

Docs

How to Use a Custom Timestamp Column

Be the first to get updates and new content

We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.

Transforming Data with ksqlDB

Hi, I'm Allison Walther with Confluent. Let's talk about transforming data with ksqlDB. KsqlDB allows us to transform our events, while creating a new stream. For example, we can change a Linux timestamp to a human-readable format, or we can remove sensitive fields from the event. When creating a new stream based on an existing one, we can pick and choose which fields we want to exist in the new stream. This allows us, for example, to provide a sanitized stream that is fit for broader use within a company. We can also use functions to modify data or create derived fields. In this example, we are using the timestamptostring function to convert the Linux epoch value to a more friendly data format, but there's really no limit to the transformations we can perform with ksqlDB. That's it for transforming data with ksqlDB, we're gonna move on to an exercise.