Getting Started with Spring for Apache Kafka ft. Viktor Gamov

Tim Berglund:

Viktor Gamov, has spent a lot of time teaching people how to use the Spring Framework and Kafka together. He's the author and instructor of the Spring for Apache Kafka course on Confluent Developer. And I've got him back on the show today to talk about that course and related things. Before we get to that, always a reminder, Streaming Audio is brought to you by Confluent Developer, that's developer.confluent.io. The one place on the web that you need to go for everything Kafka and Confluent related. If you want to take a video course, if you want some executable examples, if you want to study a library of patterns for event-driven architecture.

Tim Berglund:

There's just so much stuff there. Check it out, developer.confluent.io. And, if you do any of the examples and sign up for Confluent Cloud, to do those examples, you can use the code PODCAST100 to get an extra hundred dollars off. So check it out. Now let's get to Viktor.

Tim Berglund:

Hello and welcome to another episode of Streaming Audio. I am your host, Tim Berglund and I'm joined in the studio today by returning guest. I haven't even counted up how many times he's been on the show? Viktor Gamov. Viktor, welcome back.

Viktor Gamov:

Yes. My usual joke about long-time listener first-time caller will not work this time and it actually works just once and it's good to be here Tim. Thanks so much for having me.

Tim Berglund:

My usual joke with returning guests about this being the triumph of hope over experience. It's kind of evergreen actually but anyway, what is different now from the last time you were on the show, is that we don't work together anymore. Tell us what you're up to. Where do you work? What are you doing?

Viktor Gamov:

Yes. I am a developer advocate, with Kong, a cloud connectivity company. Where I am talking to developers about all things connectivity and how they can build resilient, scalable apps, and how we can help them with our software.

Tim Berglund:

Awesome. I believe you missed the word principal. You are a principal developer advocate. I mean, you-

Viktor Gamov:

I don't want to be sound like a-

Tim Berglund:

I said it for you though.

Viktor Gamov:

Yes. I am a principal developer advocate at Kong.

Tim Berglund:

Right. Now-

Viktor Gamov:

But still down to earth contributing as an individual, as much as I can. Meeting my people in trenches, meeting my people in conferences, not so many in conferences these days, right? But meeting my users online everywhere in the world.

Tim Berglund:

As wonderfully as you've ever done it.

Viktor Gamov:

Thank you.

Tim Berglund:

And, by the way, just as a side note listeners, we're going to talk about Viktor's Spring and Kafka course on Confluent Developer. That's what we're here to talk about today. We're going to get to that, but you know it's me and Viktor, there might be some chit chat. We're recording this at the very end of August 2021. Just last week, Viktor and I were in Krakow, Poland together at Devoxx Poland. It was the first in-person conference either one of us had done since the start of the pandemic. I think it felt good.

Viktor Gamov:

It feels good right? It felt good.

Tim Berglund:

It felt good. It just felt good. I think we can all agree.

Viktor Gamov:

And, as a matter of fact, I was actually talking about Spring and Kafka. So I think it is 100% related to the conversation of today's episode.

Tim Berglund:

Absolutely. I was at that talk. So anyway, you are the instructor of the Spring and Kafka course on Confluent Developer. This was again, insider details, folks. This was Viktor's last act as a Confluent employee heroically pushing to get this course written and-

Viktor Gamov:

It was labor of love. Come on. It was an exciting thing.

Tim Berglund:

Of course, labor of love. Love for your team and the time at the company and the technology and all that stuff. But, it was a heroic push. I watched it happen from afar and it's there and you can go watch that course. And, obviously, it's going to be prominently linked from the show notes. But, I kind of wanted to talk about Spring and Kafka course development. Just your process and your thoughts on all this. It's really just a chance to talk. So let's start with a kind of core technical issues... Spring and Kafka. What does Spring bring to the party? When is this appropriate? Why would anybody do it? Kind of give us your thoughts on that?

Viktor Gamov:

Of course, probably if people have seen me talking about Spring, I'm a huge fan. So, I didn't have any kind of child issues with Spring. When Spring was young, it hurt a lot of people and many people still have this kind of memory-

Tim Berglund:

I was there.

Viktor Gamov:

So that's why-

Tim Berglund:

I was there Gandalf, 3000 years ago.

Viktor Gamov:

Some people were there. Exactly. But it's not your grandfather or your father's Spring anymore. So it is cool. It's fast. It's super nice. And, the main goal as Spring and mainly Spring Boot these days is a main locomotion or driving base in the future, and specifically, developer experience is incredibly important for the team. People are aware about people have complaints about this Spring does that, Spring does that like too opinionated. Yes, it is too opinionated. And it's a good thing to get you going. It's probably one of the best frameworks and tools that exist today. Other frameworks, either trying to mimic what the Spring does and Spring Boot does, or they're trying to reinvent the wheel, which eventually will come into the similar places where the developers of the Spring Framework came up with.

Tim Berglund:

You either die a hero or live long enough to become kind, of the same thing as the Spring Framework, I think is what you're saying.

Viktor Gamov:

Yeah, exactly. So, not the villain. In this particular case, it's going to be a good hero. Maybe you either die as a hero or you will wait long enough until you will be invited to Marvel Cinematic Universe and you will get to reprise some different role, and everybody will forget that you have ever done, I don't know, human torch and people will remember you as a Captain America.

Tim Berglund:

Hey, you know that the first Fantastic Four movie I think was okay. I always thought-

Viktor Gamov:

It was okay.

Tim Berglund:

Jessica Alba is wonderful in a lot of roles. I think she was miscast as Sue Storm. I think it was a little bit of a miss, but she delivered on the role and I think the movie held together.

Viktor Gamov:

Yeah. I like the Mr. Fantastic. Ioan-

Tim Berglund:

Yeah.

Viktor Gamov:

He was all right.

Tim Berglund:

Again, a little young, I thought. Didn't quite nail it. Because with Reed, you sort of need that middle-aged vibe. Reed's 45 or so. And he was 30 if he was lucky. And, typical of an action movie, everybody needs to be of a certain age. And that age is young.

Viktor Gamov:

But let's agree that Julian McMahon was great as a Victor von Doom.

Tim Berglund:

Yes absolutely.

Viktor Gamov:

I think he nailed as a total villain.

Tim Berglund:

Absolutely. And, by the way, if you could have listened in to any number of Tim and Viktor one-on-ones over the years. This is maybe 20% of them right here. All right. So-

Viktor Gamov:

Talking about...so the thing is-

Tim Berglund:

People, I want to ask a question. Incase-

Viktor Gamov:

Let's talk about the opinions.

Tim Berglund:

Before we get opinions. I want to ask a question. In case you are not a part of the Java ecosystem, but you still listened to this podcast or you just have avoided Spring altogether. Could you differentiate Spring Boot and Spring Framework? People say Spring, and you said they often mean Spring Boot, which is the case, but what's the distinction?

Viktor Gamov:

The distinction is easy. So you have an engine and you have a very first car. So Spring Framework is the engine. It's a heart. It is something that drives many things, many decisions that happened inside the frameworks, and some of the frameworks that surround the Spring ecosystem. Spring Boot gives you a vehicle where you jump in and depend on your need. You might have a very fast vehicle, just like a very fast reactive microservices for the web. You will have a power of a Spring Boot as a dependence injection. And, if you want to have something more-

Tim Berglund:

Spring Framework as dependency injection?

Viktor Gamov:

Yes. Spring Framework always has been a system that allows you to do all sorts of dependence, injection things. And, some of the ways how you configure your application components. And-

Tim Berglund:

Go ahead.

Viktor Gamov:

Yeah. So if you want to go with and very, heavy, data grinding system, like a tractor, think about this. You will have all the frameworks around this, like Spring data, there's a Spring cloud data flow, data stream. So there's plenty of different frameworks that allow you to build this machinery. But as a heart, as engine as everything tied together, they tie it through the Spring Framework and Spring Boot will give you this like a starter thing. You know how to operate a new car. Spring Boot will give you this kind of feeling that you know, that this is a Spring Boot and Spring Framework is going to be an engine. And after that, the rest of the components is some additional appliances that you can put in your car and make it either lighter or make it even faster, more robust, and what have you.

Tim Berglund:

Or, bigger and heavier, but has more stuff like it's a tractor or a combine. It's not a car anymore. You can build specialized machinery. That's not just for around town. Love the analogy, but opinions, Spring opinions, and these are going to intersect with Spring's wrapping of Kafka. So take us there.

Viktor Gamov:

Yeah, exactly. So the idea of opinions is kind of... Sometimes people think, "Oh, it's too opinionated or this person too opinionated. He has opinions about everything." Which creates this-

Tim Berglund:

That's a little personal, but go on.

Viktor Gamov:

... Yeah. I know. When you're talking about the person who is opinionated or something is opinionated. You know what to expect, essentially. If you're talking about someone about politics, you pretty much know what to expect from that person. If you're talking about someone about some of the health and fitness and personal's opinion, you pretty much know what the person positioned with about something. Well, the same thing with the framework, you know what to expect. If it's opinionated, it will give you some of the stuff that gives you a quick start. But apart from being opinionated and apart from providing opinions, it's always good when the person is flexible and provides the kind of a clear view of things. So you can say, "Yes, I am opinionated about certain things. However, I'm open to hearing other people's opinions. And I am open to change my mind. If those opinions would be legitimate." Its same thing with-

Tim Berglund:

Or at least you'll give room for that other person to have his or her own opinions and sort of coexisting in the same space.

Viktor Gamov:

Yeah. Like-

Tim Berglund:

That's the best way to be opinionated.

Viktor Gamov:

... It might sound weird for you, but you kind of allow it to do this.

Tim Berglund:

Exactly. Yeah.

Viktor Gamov:

So, here's the reference. Write down in the comments if you got the reference. So same thing-

Tim Berglund:

That line right there. Let us know if you got that. Go ahead.

Viktor Gamov:

Yeah. And in the opinions of Spring, it actually has a clear path to how we could change the opinions of the Spring. If you don't like the way, how the spring does things, you will be able to change it. And, the Spring clearly identifies how we can do this. Many things that you can do there, you will be able to change. If you don't like the way, how the Spring handles, or works with your messages that you consumed or automatically consume offset or how many... If the spring behaves the same way as the GMS application behaves, you could change that. And, this is available and as a developer, you will find the ways, you don't need to change everything. You only change the things that you don't like personally, or it's somehow going as a problem with your opinions.

Viktor Gamov:

That's pretty much it. So, for the rest of the stuff, once you learn to live in this ecosystem, you will be pretty productive. And, I'm not saying happily. We're doing software development for a really long time. So, happiness is not the word that we're looking for here, but you will be able to accomplish what you want to accomplish.

Tim Berglund:

Yes. And, the goal is to fuss with the plumbing less.

Viktor Gamov:

That's the bigger thing that Spring can do for you. A lot of... And, this is kind of a mostly Spring Boot thing, because Spring Boot was created as a framework to drive this twelve-factor manifesto and drive it into production basically. Allow you to focus on things that are important for your application. And if you accept this, "I am a 12-factor app. I'm building an application that will be running in the cloud, will be running infrastructure that might go down. My application will be rescheduled. So yes, I want to build resilient applications and cloud nature of applications." You embrace this twelve-factor thingy. And, the way how the Spring configures, the way how you can override some configurations, it is more or less standard. So regardless if you're running Spring Data application or you're running Spring for Apache Kafka application, if you want to change something, I don't know.

Viktor Gamov:

Database URL in your Spring Data and bootstrap server for your Kafka application, you will be able to do this through, standard ways. For example, you can have hierarchical configurations, for example, Spring_Kafka, everything goes under this one and you will be able to change this configuration through the environment variable. You want to change this URL for the database, you're doing something similar, like spring data and you go in and change it. And, as you mentioned, as you point out you not tinkering with this plumbing for, "oh, I will write my own password for property files. I'll write my own password for the YAML file. It's a good thing is that you don't like YAML-

Tim Berglund:

[crosstalk 00:15:25] Logic for when environment variables will override things and all that. You won't write that code.

Viktor Gamov:

You just need to follow. You need to learn. What are the precedents Spring uses. You'll be fine.

Tim Berglund:

Spring famously wraps this, that, and the other piece of infrastructure. There are templates and listeners and these kinds of things for messaging frameworks and for databases. What's the Spring for Apache Kafka abstraction? What are those opinions there? And are they good opinions in your opinion?

Viktor Gamov:

Mm-hmm (affirmative). In the course that we were talking about on developer.confluent.io, I tried to provide just the bare overview of the things that are available for you. We didn't go into very details on certain things available there. During my run at Confluent, I did a lot of live streams where I drill very deeper on some of the deep cuts on the Spring Framework and integrations, based on some people's questions. People want you to say, "Hey, I have an opinion about this. And I don't like how it's done by default, Hey, I show you how we can do that." But in this course, I was trying to show how to get things done.

Viktor Gamov:

Say, you have a task, you want to produce a message to Kafka topic. So here's how we can do it. There's the KafkaTemplate. And, if you go into a little bit more in future chapters of the scores, you'll find another way. There is a Spring Cloud Streams, which is another approach, how we can solve the same task. Depends on what is closer to you. If you want to stay closer to home and Java API for Kafka is something that you enjoy. And, this is something comfortable Spring for Apache Kafka gives you a very thin wrapper that allows you to focus on that type of APIs, whereas-

Tim Berglund:

Wrapper around the producer and consumer.

Viktor Gamov:

Exactly. With the Spring Cloud Stream, you'll have more opinion, again, a more opinionated way where you're focusing on actual logic. And, after that plumbing and the framework and the conventions will fulfill the gaps. You're defining the function that will produce data and after that, the underlying technology will figure out how to take this data and turn it into something that Kafka will understand. Well, I'm talking about the binders for Kafka. Those also like much higher level obstruction and mostly, maybe people will appreciate it if they will be linked in some sort of a data pipelines based on Java code and the functional code and maybe even kind of a running function as a service, which framework will handle some of the plumbing and how the things will be connected to underlying Kafka search.

Tim Berglund:

Okay. Other parts of Spring will abstract away your functions and service stuff. Which is, I mean-

Viktor Gamov:

so-

Tim Berglund:

The benefit of Spring, right, is that there's a kind of a wrapper and an abstraction for just about everything-

Viktor Gamov:

Which is-

Tim Berglund:

Once you-

Viktor Gamov:

Sometimes it can be very overwhelming, but there's the interesting thing. The spring itself, even though they accomplished a lot of things in covering different frameworks, it does not stay the same over time. So I was not joking when I said that it's not like your father's or your grandfather's framework, a lot of things changed. And, based on understanding and general consensus among developers, I'll give you a quick example with a framework that I like and enjoy, the Spring Cloud Stream. So a lot of work in Spring Cloud Stream was influenced by Spring messaging framework that was using the messaging system, like a JMS as a transport.

Viktor Gamov:

So, that's why on the very early versions of this, you will find actually not the messaging, but rather than Spring Integration, which is was genetic framework that allowed you to integrate different systems. Before say, Kafka, become our kind of a defacto integration thing. Different people were building different pipelines differently. It might be databases. They might be some GMS systems. So they will try to... Spring Integration was a kind of implementation of some of the ideas that were described in the Enterprise Integration Patterns.

Tim Berglund:

Which is back there somewhere.

Viktor Gamov:

But, over time those abstractions become not convoluted but at least they become something that will rather confuse people than give them some good. So that's why with the Spring Cloud Streams version 3, they, went to the point where, "Okay, so let's focus on the logic. Let's focus on functions rather than plumbing." And we, as a framework was smart enough to figure out how to take your bites and turn them into Kafka messages. It's the same, same thing all the time. We will just need to know what to invoke in order to get this data in case of a producer or we need to know where we can ship this data or this bites in case of the consumer. So in this case, you defined a supplier in case of producer [inaudible 00:21:11] a Java supplier, you defined a consumer as a Java consumer as a receiver of this data from the framework and this convention of a configuration function-based approach changed the way, how you can think about your application. You focus only on the logic and the plumbing will be done.

Tim Berglund:

Got it. In the springy way of annotating. I guess these days it is really mostly-

Viktor Gamov:

That's a neat part. You don't even need to annotate these days. Everything comes with... There's one annotation is there is a bean, which is a kind of standard way how you can tell that this is the component that belongs to your application logic, but you don't need to specify where this data comes from. You can fully externalize it. And, based on the things that I said before, those configurations can be changed. You have a YAML-

Tim Berglund:

[crosstalk 00:22:08] like YAML and environment variables and all that.

Viktor Gamov:

Yeah, exactly.

Tim Berglund:

You still have an application guide-

Viktor Gamov:

You have your YAML.

Tim Berglund:

That's true.

Viktor Gamov:

application.yaml or application.properties, and you can go ahead and after that, just override this, whenever you move to another environment, just like change this environment variable and your application logic and all this wiring will stay the same. Just the only thing changes is the bootstrap URL for... Where is your Kafka case deployed? Basically.

Tim Berglund:

Of course, I burying those childhood wounds will still always want to say application.xml. I don't think it'll ever leave me.

Viktor Gamov:

I believe you still will be able to do this.

Tim Berglund:

Oh no, you can.

Viktor Gamov:

But probably you still can do this.

Tim Berglund:

Just one does not simply... So when you moving to your actual design of this course, when you want to sit down because you know this very well, you did a year's worth of live streams episodes, not all on Spring, but that was very much a recurring theme. So it's a thing you do a lot. So when you sit down and you want to build an hour or an hour and a half of video content, introducing someone to how to do Spring and Kafka, how to build things with Kafka using Spring, kind of assuming certainly that they've got some Java, they've probably got some familiarity, the Spring Framework. How did you break that down? I'm kind of asking you what's the structure of the course, but I'm as much asking you, what's your thought process when you do that kind of thing?

Viktor Gamov:

When I thinking about how I will teach something or like transfer my knowledge to someone with the most productive way or most efficient way, so this person, they would pick up this quickly. I was trying to think from the point how I would love to learn things about a certain subject. So, and the couple use cases will help. Over the time working with customers and working with developers, I understand the first thing people want to see, it's just like connect to some Kafka and see that data is flowing and have this kind of a nice and fuzzy feeling that Ooh, data is coming. And, this is the first thing that people want to do. Like, okay. So I want to try this. And, in order to do this, you need to have some data, some there, I didn't want to kind of convolute this or overload this example by kind of I don't know, like bringing connect into play, even though that would be fun because I don't need to write a producer connect will be pumping some data and I'll have a consumer.

Viktor Gamov:

So, in this case, I decide, okay. Before I start reading data, I need to bring some data in. I need to generate some data. That's why usually I like to start with, "okay, so you right there, next thing we'll see how you can retrieve this data." And, always tools that provide some of the UI. So Confluent Cloud has a nice UI where you can see that data is actually flowing there, so you don't need to write your consumer per se, in order to see this. So, that's why first thing is, one will do is just produce a message, hello, world message to Kafka. And, after that UI will be able to show all the things. After that consumer and after the processing part is also important apart from being just like a simple consumer that read the data and doing something [inaudible 00:25:52]

Viktor Gamov:

I wanted people to see that Kafka Streams integrates very nicely with Spring Boot mentality, Spring Boot will handle Kafka Streams, it doesn't have any infrastructure to read configuration files or whatnot. Sometimes people complain about this is that the Kafka Streams doesn't have it. And, there should be some opinionated way. However, Kafka and Kafka tools around this, It's like a UNIX tool. If you want to build something around this, you want to have some configuration for this. You will be able to do this thanks to the frameworks that are available. And, this is where Spring comes into play in saying, "Hey, I can save the day. I know how you can configure your Kafka Streams-

Tim Berglund:

"I've got all kinds of opinions about this." Where Kafka Streams is wanting to be out of the business of having that kind of opinion.

Viktor Gamov:

Exactly. And as a developer, just want to have a configured instance of your property file that you will or even you want to have this streams, builder that already preconfigured. So you can have defined your topology. And this topology would be injected into... StreamsBuilder would be created for you because it's just a few more lines of code. Another thing is that usually, people don't know what to do with streams, the Kafka Streams instance, because you need to start this and when you stop it, how you will clean up it Spring helps you to handle this and you only focus on your logic. So, the next step is we have a producer, we got a consumer, we have a stream to process this. And, you will see it in the course, if you will take this, which you should.

Tim Berglund:

Which you should.

Viktor Gamov:

That's kind of idea where we going from something that you have a need and you go on the journey.

Tim Berglund:

From something that you know plus a need that you have, something you already know plus something you want, but don't have and that brings people kind of into the educational process of the course. That's a good way for it to do a talk-

Viktor Gamov:

Exactly. So, the storytelling and the plot design development and the things there's like a concept of the hero journey. I like to follow this one. I like that. It's a kind of a hero journey plus there's like a hero cycle. I don't how it got like eight steps then Harmon's cycle of story and things like that. I want you to have a full-cycle return back home changed. However, this change will give you some new expertise that allows you to move in another cycle in another cycle.

Tim Berglund:

Exactly.

Viktor Gamov:

So, that's why the idea was approached more from the perspective of how to teach this and how to help people to... In inviting them for a journey to-

Tim Berglund:

Yes. A journey that will change them in a positive way. Send them back home with new capabilities. Could not have said it better myself.

Viktor Gamov:

Since I use this opportu-

Tim Berglund:

[inaudible 00:29:12].

Viktor Gamov:

I want to use this opportunity also since we have listeners and we're talking about some of this stuff about learning Kafka. I want to say that it's almost September and meaning that Manning's Kafka in Action book is coming to book shoulders.

Tim Berglund:

Yes.

Viktor Gamov:

And, this is a book where I quarter with my colleagues, Dylan Scott and Dave Klein from Counseling and Dylan Scott from the community. He started this project and we help him to finish this journey. So for listeners, we'll put some code there so you can get the early access program and get some discounts. If you are interested in this type of reading. And again, the idea of this book, there's like plenty of good Kafka books that go a little bit like inside things and talk about some of the cool, exciting guts of this exciting system would try to approach with the more practical things. What kind of practical things do you need to know. And in order to use Kafka like efficiently, so Kafka in Action coming to your book stores. I will sign every copy wherever you will find me. I will do that. So-

Tim Berglund:

You can-

Viktor Gamov:

Thanks for this opportunity to-

Tim Berglund:

You can take his word for that. He will do that.

Viktor Gamov:

Yeah. And, I was extremely excited that I had this opportunity to share some of my knowledge about things. So, I was super excited when the course was released earlier this year.

Tim Berglund:

My guest today has been Viktor Gamov. Viktor, thanks for being a part of Streaming Audio.

Viktor Gamov:

It was a pleasure Tim as always, and it was Viktor Gamov and as always have a nice day.

Tim Berglund:

And there you have it. Thanks for listening to this episode. Now, some important details before you go Streaming Audio is brought to you by Confluent Developer, that's developer.confluent.io, a website dedicated to helping you learn Kafka, Confluent, and everything in the broader event streaming ecosystem. We've got free video courses, a library of event-driven architecture design patterns, executable tutorials, covering ksqlDB, Kafka Streams, and core Kafka APIs. There's even an index of episodes of this podcast. So if you take a course on Confluent Developer, you'll have the chance to use Confluent Cloud. When you sign up, use the code PODCAST100 to get an extra $100 of free Confluent Cloud usage. Anyway, as always, I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can always reach out to me @tlberglund on Twitter.

Tim Berglund:

That's T-L-B-E-R-G-L-U-N-D. Or you can leave a comment on the YouTube video if you're watching and not just listening or reach out in our community slack or forum, both are linked in the show notes. And, while you're at it, please subscribe to our YouTube channel and to this podcast, wherever fine podcasts are sold. And, if you subscribe through Apple podcasts, be sure to leave us a review there. That helps other people discover us, which we think is a good thing. So, thanks for your support, and we'll see you next time.