Viktor Gamov, has spent a lot of time teaching people how to use the Spring Framework and Kafka together. He's the author and instructor of the Spring for Apache Kafka course on Confluent Developer. And I've got him back on the show today to talk about that course and related things. Before we get to that, always a reminder, Streaming Audio is brought to you by Confluent Developer, that's developer.confluent.io. The one place on the web that you need to go for everything Kafka and Confluent related. If you want to take a video course, if you want some executable examples, if you want to study a library of patterns for event-driven architecture.
There's just so much stuff there. Check it out, developer.confluent.io. And, if you do any of the examples and sign up for Confluent Cloud, to do those examples, you can use the code PODCAST100 to get an extra hundred dollars off. So check it out. Now let's get to Viktor.
Hello and welcome to another episode of Streaming Audio. I am your host, Tim Berglund and I'm joined in the studio today by returning guest. I haven't even counted up how many times he's been on the show? Viktor Gamov. Viktor, welcome back.
Yes. My usual joke about long-time listener first-time caller will not work this time and it actually works just once and it's good to be here Tim. Thanks so much for having me.
My usual joke with returning guests about this being the triumph of hope over experience. It's kind of evergreen actually but anyway, what is different now from the last time you were on the show, is that we don't work together anymore. Tell us what you're up to. Where do you work? What are you doing?
Yes. I am a developer advocate, with Kong, a cloud connectivity company. Where I am talking to developers about all things connectivity and how they can build resilient, scalable apps, and how we can help them with our software.
Awesome. I believe you missed the word principal. You are a principal developer advocate. I mean, you-
I don't want to be sound like a-
I said it for you though.
Yes. I am a principal developer advocate at Kong.
But still down to earth contributing as an individual, as much as I can. Meeting my people in trenches, meeting my people in conferences, not so many in conferences these days, right? But meeting my users online everywhere in the world.
As wonderfully as you've ever done it.
And, by the way, just as a side note listeners, we're going to talk about Viktor's Spring and Kafka course on Confluent Developer. That's what we're here to talk about today. We're going to get to that, but you know it's me and Viktor, there might be some chit chat. We're recording this at the very end of August 2021. Just last week, Viktor and I were in Krakow, Poland together at Devoxx Poland. It was the first in-person conference either one of us had done since the start of the pandemic. I think it felt good.
It feels good right? It felt good.
It felt good. It just felt good. I think we can all agree.
And, as a matter of fact, I was actually talking about Spring and Kafka. So I think it is 100% related to the conversation of today's episode.
Absolutely. I was at that talk. So anyway, you are the instructor of the Spring and Kafka course on Confluent Developer. This was again, insider details, folks. This was Viktor's last act as a Confluent employee heroically pushing to get this course written and-
It was labor of love. Come on. It was an exciting thing.
Of course, labor of love. Love for your team and the time at the company and the technology and all that stuff. But, it was a heroic push. I watched it happen from afar and it's there and you can go watch that course. And, obviously, it's going to be prominently linked from the show notes. But, I kind of wanted to talk about Spring and Kafka course development. Just your process and your thoughts on all this. It's really just a chance to talk. So let's start with a kind of core technical issues... Spring and Kafka. What does Spring bring to the party? When is this appropriate? Why would anybody do it? Kind of give us your thoughts on that?
Of course, probably if people have seen me talking about Spring, I'm a huge fan. So, I didn't have any kind of child issues with Spring. When Spring was young, it hurt a lot of people and many people still have this kind of memory-
I was there.
So that's why-
I was there Gandalf, 3000 years ago.
Some people were there. Exactly. But it's not your grandfather or your father's Spring anymore. So it is cool. It's fast. It's super nice. And, the main goal as Spring and mainly Spring Boot these days is a main locomotion or driving base in the future, and specifically, developer experience is incredibly important for the team. People are aware about people have complaints about this Spring does that, Spring does that like too opinionated. Yes, it is too opinionated. And it's a good thing to get you going. It's probably one of the best frameworks and tools that exist today. Other frameworks, either trying to mimic what the Spring does and Spring Boot does, or they're trying to reinvent the wheel, which eventually will come into the similar places where the developers of the Spring Framework came up with.
You either die a hero or live long enough to become kind, of the same thing as the Spring Framework, I think is what you're saying.
Yeah, exactly. So, not the villain. In this particular case, it's going to be a good hero. Maybe you either die as a hero or you will wait long enough until you will be invited to Marvel Cinematic Universe and you will get to reprise some different role, and everybody will forget that you have ever done, I don't know, human torch and people will remember you as a Captain America.
Hey, you know that the first Fantastic Four movie I think was okay. I always thought-
It was okay.
Jessica Alba is wonderful in a lot of roles. I think she was miscast as Sue Storm. I think it was a little bit of a miss, but she delivered on the role and I think the movie held together.
Yeah. I like the Mr. Fantastic. Ioan-
He was all right.
Again, a little young, I thought. Didn't quite nail it. Because with Reed, you sort of need that middle-aged vibe. Reed's 45 or so. And he was 30 if he was lucky. And, typical of an action movie, everybody needs to be of a certain age. And that age is young.
But let's agree that Julian McMahon was great as a Victor von Doom.
I think he nailed as a total villain.
Absolutely. And, by the way, if you could have listened in to any number of Tim and Viktor one-on-ones over the years. This is maybe 20% of them right here. All right. So-
Talking about...so the thing is-
People, I want to ask a question. Incase-
Let's talk about the opinions.
Before we get opinions. I want to ask a question. In case you are not a part of the Java ecosystem, but you still listened to this podcast or you just have avoided Spring altogether. Could you differentiate Spring Boot and Spring Framework? People say Spring, and you said they often mean Spring Boot, which is the case, but what's the distinction?
The distinction is easy. So you have an engine and you have a very first car. So Spring Framework is the engine. It's a heart. It is something that drives many things, many decisions that happened inside the frameworks, and some of the frameworks that surround the Spring ecosystem. Spring Boot gives you a vehicle where you jump in and depend on your need. You might have a very fast vehicle, just like a very fast reactive microservices for the web. You will have a power of a Spring Boot as a dependence injection. And, if you want to have something more-
Spring Framework as dependency injection?
Yes. Spring Framework always has been a system that allows you to do all sorts of dependence, injection things. And, some of the ways how you configure your application components. And-
Yeah. So if you want to go with and very, heavy, data grinding system, like a tractor, think about this. You will have all the frameworks around this, like Spring data, there's a Spring cloud data flow, data stream. So there's plenty of different frameworks that allow you to build this machinery. But as a heart, as engine as everything tied together, they tie it through the Spring Framework and Spring Boot will give you this like a starter thing. You know how to operate a new car. Spring Boot will give you this kind of feeling that you know, that this is a Spring Boot and Spring Framework is going to be an engine. And after that, the rest of the components is some additional appliances that you can put in your car and make it either lighter or make it even faster, more robust, and what have you.
Or, bigger and heavier, but has more stuff like it's a tractor or a combine. It's not a car anymore. You can build specialized machinery. That's not just for around town. Love the analogy, but opinions, Spring opinions, and these are going to intersect with Spring's wrapping of Kafka. So take us there.
Yeah, exactly. So the idea of opinions is kind of... Sometimes people think, "Oh, it's too opinionated or this person too opinionated. He has opinions about everything." Which creates this-
That's a little personal, but go on.
... Yeah. I know. When you're talking about the person who is opinionated or something is opinionated. You know what to expect, essentially. If you're talking about someone about politics, you pretty much know what to expect from that person. If you're talking about someone about some of the health and fitness and personal's opinion, you pretty much know what the person positioned with about something. Well, the same thing with the framework, you know what to expect. If it's opinionated, it will give you some of the stuff that gives you a quick start. But apart from being opinionated and apart from providing opinions, it's always good when the person is flexible and provides the kind of a clear view of things. So you can say, "Yes, I am opinionated about certain things. However, I'm open to hearing other people's opinions. And I am open to change my mind. If those opinions would be legitimate." Its same thing with-
Or at least you'll give room for that other person to have his or her own opinions and sort of coexisting in the same space.
That's the best way to be opinionated.
... It might sound weird for you, but you kind of allow it to do this.
So, here's the reference. Write down in the comments if you got the reference. So same thing-
That line right there. Let us know if you got that. Go ahead.
Yeah. And in the opinions of Spring, it actually has a clear path to how we could change the opinions of the Spring. If you don't like the way, how the spring does things, you will be able to change it. And, the Spring clearly identifies how we can do this. Many things that you can do there, you will be able to change. If you don't like the way, how the Spring handles, or works with your messages that you consumed or automatically consume offset or how many... If the spring behaves the same way as the GMS application behaves, you could change that. And, this is available and as a developer, you will find the ways, you don't need to change everything. You only change the things that you don't like personally, or it's somehow going as a problem with your opinions.
That's pretty much it. So, for the rest of the stuff, once you learn to live in this ecosystem, you will be pretty productive. And, I'm not saying happily. We're doing software development for a really long time. So, happiness is not the word that we're looking for here, but you will be able to accomplish what you want to accomplish.
Yes. And, the goal is to fuss with the plumbing less.
That's the bigger thing that Spring can do for you. A lot of... And, this is kind of a mostly Spring Boot thing, because Spring Boot was created as a framework to drive this twelve-factor manifesto and drive it into production basically. Allow you to focus on things that are important for your application. And if you accept this, "I am a 12-factor app. I'm building an application that will be running in the cloud, will be running infrastructure that might go down. My application will be rescheduled. So yes, I want to build resilient applications and cloud nature of applications." You embrace this twelve-factor thingy. And, the way how the Spring configures, the way how you can override some configurations, it is more or less standard. So regardless if you're running Spring Data application or you're running Spring for Apache Kafka application, if you want to change something, I don't know.
Database URL in your Spring Data and bootstrap server for your Kafka application, you will be able to do this through, standard ways. For example, you can have hierarchical configurations, for example, Spring_Kafka, everything goes under this one and you will be able to change this configuration through the environment variable. You want to change this URL for the database, you're doing something similar, like spring data and you go in and change it. And, as you mentioned, as you point out you not tinkering with this plumbing for, "oh, I will write my own password for property files. I'll write my own password for the YAML file. It's a good thing is that you don't like YAML-
[crosstalk 00:15:25] Logic for when environment variables will override things and all that. You won't write that code.
You just need to follow. You need to learn. What are the precedents Spring uses. You'll be fine.
Spring famously wraps this, that, and the other piece of infrastructure. There are templates and listeners and these kinds of things for messaging frameworks and for databases. What's the Spring for Apache Kafka abstraction? What are those opinions there? And are they good opinions in your opinion?
Mm-hmm (affirmative). In the course that we were talking about on developer.confluent.io, I tried to provide just the bare overview of the things that are available for you. We didn't go into very details on certain things available there. During my run at Confluent, I did a lot of live streams where I drill very deeper on some of the deep cuts on the Spring Framework and integrations, based on some people's questions. People want you to say, "Hey, I have an opinion about this. And I don't like how it's done by default, Hey, I show you how we can do that." But in this course, I was trying to show how to get things done.
Say, you have a task, you want to produce a message to Kafka topic. So here's how we can do it. There's the KafkaTemplate. And, if you go into a little bit more in future chapters of the scores, you'll find another way. There is a Spring Cloud Streams, which is another approach, how we can solve the same task. Depends on what is closer to you. If you want to stay closer to home and Java API for Kafka is something that you enjoy. And, this is something comfortable Spring for Apache Kafka gives you a very thin wrapper that allows you to focus on that type of APIs, whereas-
Wrapper around the producer and consumer.
Exactly. With the Spring Cloud Stream, you'll have more opinion, again, a more opinionated way where you're focusing on actual logic. And, after that plumbing and the framework and the conventions will fulfill the gaps. You're defining the function that will produce data and after that, the underlying technology will figure out how to take this data and turn it into something that Kafka will understand. Well, I'm talking about the binders for Kafka. Those also like much higher level obstruction and mostly, maybe people will appreciate it if they will be linked in some sort of a data pipelines based on Java code and the functional code and maybe even kind of a running function as a service, which framework will handle some of the plumbing and how the things will be connected to underlying Kafka search.
Okay. Other parts of Spring will abstract away your functions and service stuff. Which is, I mean-
The benefit of Spring, right, is that there's a kind of a wrapper and an abstraction for just about everything-
Sometimes it can be very overwhelming, but there's the interesting thing. The spring itself, even though they accomplished a lot of things in covering different frameworks, it does not stay the same over time. So I was not joking when I said that it's not like your father's or your grandfather's framework, a lot of things changed. And, based on understanding and general consensus among developers, I'll give you a quick example with a framework that I like and enjoy, the Spring Cloud Stream. So a lot of work in Spring Cloud Stream was influenced by Spring messaging framework that was using the messaging system, like a JMS as a transport.
So, that's why on the very early versions of this, you will find actually not the messaging, but rather than Spring Integration, which is was genetic framework that allowed you to integrate different systems. Before say, Kafka, become our kind of a defacto integration thing. Different people were building different pipelines differently. It might be databases. They might be some GMS systems. So they will try to... Spring Integration was a kind of implementation of some of the ideas that were described in the Enterprise Integration Patterns.
Which is back there somewhere.
But, over time those abstractions become not convoluted but at least they become something that will rather confuse people than give them some good. So that's why with the Spring Cloud Streams version 3, they, went to the point where, "Okay, so let's focus on the logic. Let's focus on functions rather than plumbing." And we, as a framework was smart enough to figure out how to take your bites and turn them into Kafka messages. It's the same, same thing all the time. We will just need to know what to invoke in order to get this data in case of a producer or we need to know where we can ship this data or this bites in case of the consumer. So in this case, you defined a supplier in case of producer [inaudible 00:21:11] a Java supplier, you defined a consumer as a Java consumer as a receiver of this data from the framework and this convention of a configuration function-based approach changed the way, how you can think about your application. You focus only on the logic and the plumbing will be done.
Got it. In the springy way of annotating. I guess these days it is really mostly-
That's a neat part. You don't even need to annotate these days. Everything comes with... There's one annotation is there is a bean, which is a kind of standard way how you can tell that this is the component that belongs to your application logic, but you don't need to specify where this data comes from. You can fully externalize it. And, based on the things that I said before, those configurations can be changed. You have a YAML-
[crosstalk 00:22:08] like YAML and environment variables and all that.
You still have an application guide-
You have your YAML.
application.yaml or application.properties, and you can go ahead and after that, just override this, whenever you move to another environment, just like change this environment variable and your application logic and all this wiring will stay the same. Just the only thing changes is the bootstrap URL for... Where is your Kafka case deployed? Basically.
Of course, I burying those childhood wounds will still always want to say application.xml. I don't think it'll ever leave me.
I believe you still will be able to do this.
Oh no, you can.
But probably you still can do this.
Just one does not simply... So when you moving to your actual design of this course, when you want to sit down because you know this very well, you did a year's worth of live streams episodes, not all on Spring, but that was very much a recurring theme. So it's a thing you do a lot. So when you sit down and you want to build an hour or an hour and a half of video content, introducing someone to how to do Spring and Kafka, how to build things with Kafka using Spring, kind of assuming certainly that they've got some Java, they've probably got some familiarity, the Spring Framework. How did you break that down? I'm kind of asking you what's the structure of the course, but I'm as much asking you, what's your thought process when you do that kind of thing?
When I thinking about how I will teach something or like transfer my knowledge to someone with the most productive way or most efficient way, so this person, they would pick up this quickly. I was trying to think from the point how I would love to learn things about a certain subject. So, and the couple use cases will help. Over the time working with customers and working with developers, I understand the first thing people want to see, it's just like connect to some Kafka and see that data is flowing and have this kind of a nice and fuzzy feeling that Ooh, data is coming. And, this is the first thing that people want to do. Like, okay. So I want to try this. And, in order to do this, you need to have some data, some there, I didn't want to kind of convolute this or overload this example by kind of I don't know, like bringing connect into play, even though that would be fun because I don't need to write a producer connect will be pumping some data and I'll have a consumer.
So, in this case, I decide, okay. Before I start reading data, I need to bring some data in. I need to generate some data. That's why usually I like to start with, "okay, so you right there, next thing we'll see how you can retrieve this data." And, always tools that provide some of the UI. So Confluent Cloud has a nice UI where you can see that data is actually flowing there, so you don't need to write your consumer per se, in order to see this. So, that's why first thing is, one will do is just produce a message, hello, world message to Kafka. And, after that UI will be able to show all the things. After that consumer and after the processing part is also important apart from being just like a simple consumer that read the data and doing something [inaudible 00:25:52]
I wanted people to see that Kafka Streams integrates very nicely with Spring Boot mentality, Spring Boot will handle Kafka Streams, it doesn't have any infrastructure to read configuration files or whatnot. Sometimes people complain about this is that the Kafka Streams doesn't have it. And, there should be some opinionated way. However, Kafka and Kafka tools around this, It's like a UNIX tool. If you want to build something around this, you want to have some configuration for this. You will be able to do this thanks to the frameworks that are available. And, this is where Spring comes into play in saying, "Hey, I can save the day. I know how you can configure your Kafka Streams-
"I've got all kinds of opinions about this." Where Kafka Streams is wanting to be out of the business of having that kind of opinion.
Exactly. And as a developer, just want to have a configured instance of your property file that you will or even you want to have this streams, builder that already preconfigured. So you can have defined your topology. And this topology would be injected into... StreamsBuilder would be created for you because it's just a few more lines of code. Another thing is that usually, people don't know what to do with streams, the Kafka Streams instance, because you need to start this and when you stop it, how you will clean up it Spring helps you to handle this and you only focus on your logic. So, the next step is we have a producer, we got a consumer, we have a stream to process this. And, you will see it in the course, if you will take this, which you should.
Which you should.
That's kind of idea where we going from something that you have a need and you go on the journey.
From something that you know plus a need that you have, something you already know plus something you want, but don't have and that brings people kind of into the educational process of the course. That's a good way for it to do a talk-
Exactly. So, the storytelling and the plot design development and the things there's like a concept of the hero journey. I like to follow this one. I like that. It's a kind of a hero journey plus there's like a hero cycle. I don't how it got like eight steps then Harmon's cycle of story and things like that. I want you to have a full-cycle return back home changed. However, this change will give you some new expertise that allows you to move in another cycle in another cycle.
So, that's why the idea was approached more from the perspective of how to teach this and how to help people to... In inviting them for a journey to-
Yes. A journey that will change them in a positive way. Send them back home with new capabilities. Could not have said it better myself.
Since I use this opportu-
I want to use this opportunity also since we have listeners and we're talking about some of this stuff about learning Kafka. I want to say that it's almost September and meaning that Manning's Kafka in Action book is coming to book shoulders.
And, this is a book where I quarter with my colleagues, Dylan Scott and Dave Klein from Counseling and Dylan Scott from the community. He started this project and we help him to finish this journey. So for listeners, we'll put some code there so you can get the early access program and get some discounts. If you are interested in this type of reading. And again, the idea of this book, there's like plenty of good Kafka books that go a little bit like inside things and talk about some of the cool, exciting guts of this exciting system would try to approach with the more practical things. What kind of practical things do you need to know. And in order to use Kafka like efficiently, so Kafka in Action coming to your book stores. I will sign every copy wherever you will find me. I will do that. So-
Thanks for this opportunity to-
You can take his word for that. He will do that.
Yeah. And, I was extremely excited that I had this opportunity to share some of my knowledge about things. So, I was super excited when the course was released earlier this year.
My guest today has been Viktor Gamov. Viktor, thanks for being a part of Streaming Audio.
It was a pleasure Tim as always, and it was Viktor Gamov and as always have a nice day.
And there you have it. Thanks for listening to this episode. Now, some important details before you go Streaming Audio is brought to you by Confluent Developer, that's developer.confluent.io, a website dedicated to helping you learn Kafka, Confluent, and everything in the broader event streaming ecosystem. We've got free video courses, a library of event-driven architecture design patterns, executable tutorials, covering ksqlDB, Kafka Streams, and core Kafka APIs. There's even an index of episodes of this podcast. So if you take a course on Confluent Developer, you'll have the chance to use Confluent Cloud. When you sign up, use the code PODCAST100 to get an extra $100 of free Confluent Cloud usage. Anyway, as always, I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can always reach out to me @tlberglund on Twitter.
That's T-L-B-E-R-G-L-U-N-D. Or you can leave a comment on the YouTube video if you're watching and not just listening or reach out in our community slack or forum, both are linked in the show notes. And, while you're at it, please subscribe to our YouTube channel and to this podcast, wherever fine podcasts are sold. And, if you subscribe through Apple podcasts, be sure to leave us a review there. That helps other people discover us, which we think is a good thing. So, thanks for your support, and we'll see you next time.
What’s the distinction between the Spring Framework and Spring Boot? If you are building a car, the Spring Framework is the engine while Spring Boot gives you the vehicle that you ride in. With experience teaching and answering questions on how to use Spring and Apache Kafka® together, Viktor Gamov (Principal Developer Advocate, Kong) designed a free course on Confluent Developer and previews it in this episode. Not only this, but he also explains why the opinionated Spring Framework would be a good hero in Marvel.
Spring is an ever-evolving framework that embraces modern, cloud-native technologies with cross-language options, such as Kotlin integration. Unlike its predecessors, the Spring Framework supports a modern version of Java and the requirements of the Twelve-Factor App manifesto for you to move an application between environments without changing the code. With that engine in place, Spring Boot introduces a microservices architecture. Spring Boot contains databases and messaging systems integrations, reducing development time and increasing overall productivity.
Spring for Apache Kafka applies best practices of the Spring community to the Kafka ecosystem, including features that abstract away infrastructure code for you to focus on programming logic that is important for your application. Spring for Apache Kafka provides a wrapper around the producer and consumer to ease Kafka configuration with APIs, including KafkaTemplate, MessageListenerContainer, @KafkaListener, and TopicBuilder.
The Spring Framework and Apache Kafka course will equip you with the knowledge you need in order to build event-driven microservices using Spring and Kafka on Confluent Cloud. Tim and Viktor also discuss Spring Cloud Stream as well as Spring Boot integration with Kafka Streams and more.
If there's something you want to know about Apache Kafka, Confluent or event streaming, please send us an email with your question and we'll hope to answer it on the next episode of Ask Confluent.Email Us