Building an Apache Kafka Center of Excellence Within Your Organization ft. Neil Buesing

Tim Berglund (00:00):

My friend Neil Buesing is a Kafka community stalwart. And he also spends his days building things out of Kafka and helping others to do the same. I talked to him today about how a robust interest in event streaming affects a whole consulting practice and how Centers of Excellence, I know, sounds enterprisey, but they are a real and really good thing, how they're something you might want to try to be involved with as your Kafka career grows. It's all on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud.

Tim Berglund (00:38):

Hello, and welcome to another episode of Streaming Audio. I am as always your host, Tim Berglund. I'm still Tim Berglund. That is just a persistent fact of my world that has not changed. And I am happy today to be joined in the virtual studio by my friend Neil Buesing. Neil is vice president of streaming technologies at Object Partners. Object Partners is a consulting company that happens to be a Confluent partner. We work together helping people deploy Kafka Confluent platform. And Neil's also a frequent conference speaker, Kafka summit speaker, outstanding community member, Confluent community catalyst. He's all these things. Neil, welcome to the show.

Neil Buesing (01:28):

Thanks, Tim. Thanks for having me. Really glad to be here.

Tim Berglund (01:32):

You bet. Now tell us a little bit about what a vice president of streaming technologies does, given that you work at a consulting company. So what is your work like?

Neil Buesing (01:43):

Well, there's a lot of variation to the work at Object Partners. We do full stack development from mobile apps to front-end apps. But my focus has really been around the streaming technologies, which, let's face it, in the last few years have been all things, Apache Kafka, for us and Confluent Enterprise. So we work with clients to modernize their platforms to get them into a streaming event-based application and development to really modernize their platform and to make it, I guess, more, what's the right word? Eventing, more reactionary, more allowing the system to grow and mature in that way.

Tim Berglund (02:32):

Cool. And give us a feel for Object Partners? How big is the group? And how many consultants do you have an opportunity to influence on a regular basis?

Neil Buesing (02:44):

Yeah, Object Partners has been around for 24 years. I've been with them for 20. And we have our primary office here in Minneapolis, Minnesota, in the United States. And then our secondary office is in Omaha, Nebraska, which has been around for a good chunk of that time now, 14 years or so. And our focus of about 200 now consultants, in like I said, the full technology. Our bread and butter have been JVM space development. We've gone down the Groovy Grails path when that was hot. We do mobile and both iOS and Android development, a lot of front-end development, and a lot of core Java framework development using RESTful Architecture, a lot of the keywords you can speak of, and in the last few years, eventing and event driven systems.

Tim Berglund (03:42):

I imagine there have been one or two Spring applications deployed by-

Neil Buesing (03:46):

This is the Midwest, so we do a lot of Spring.

Tim Berglund (03:48):

Yes.

Neil Buesing (03:49):

I guess it's, I consulted in a variety of clients, it seems like the more you go to the coast, the more alternate languages there are. But we're starting to see Go pop up here and there. But yes, our primary language is Java so our primary platform is Spring.

Tim Berglund (04:07):

Cool. That sounds like a place where I'd be technologically very at home. So all-

Neil Buesing (04:11):

Yeah.

Tim Berglund (04:12):

... good stuff. So when you work with your clients, I want to talk today about the notion of building a Center of Excellence or a COE. Tell us what a COE is and if I could ask this in a challenging way, why our listeners care what a COE is.

Neil Buesing (04:34):

Well, a COE stands for Center of Excellence, and it's been around for as long as I've done consulting, it's been to serve up the job of framework code. So 10, 15 years ago, I would work with the Center of Excellences, primarily focused on how do you deploy your application? How do you build the application? How do you handle authentication? So are we going to use JWT tokens? What kind of SSL certs are we going to use? Who signs those certs? These are all things that a client needs to wrangle in and control for other departments within their organization. So they build a center of excellence to make sure there's a standard, as well as minimize the variations of software and nuances they would need to support.

Neil Buesing (05:23):

Well, Kafka, Apache Kafka has a lot of complexity to it. And people need to start wrangling that in so their teams can be successful, but also, so they can be successful in trying to manage all these teams. And when open source became a dominant factor in software development, a lot of companies said, "Well, we need to start controlling which versions or which libraries do people bring in." And there's a sense that with Kafka, there's so much you can do that they want to get people on the path that allows them to reduce the amount of inconsistencies or things that they're building that just requires them more work to maintain. So it's companies trying to reduce the footprint and to make their life manageable.

Tim Berglund (06:17):

Cool. And I'm interested, so I've never thought of a COE in this way. But you talked at first about Java Centers of Excellence in the good old days. That doesn't sound like a thing that would exist now. So is the Center of Excellence, a mechanism that is somehow transitional in nature, like when we're on a wave of adoption, maybe there's a period of a decade where there are COEs around a particular technology, but once it becomes mature, if you work at a client company of other partners, you probably know Java, if you're a developer. You can probably drive a build tool. Do you think of them that way?

Neil Buesing (07:05):

I wasn't thinking that too much. But now looking back at the last 15 years, there really was a point of not hearing so much about having that center. And like you said, the technology is mature, the understanding of it has been there, frameworks have adopted standards. So if you bring in Spring or Spring Boot, you have a set of standards you're already following unless you're deviating from the norm. So those tend to be a little less. I would say DevOps has definitely impacted that as more teams are able to do their own operations, they're given more freedom to do what they want, because you're not handing it off to others to maintain. So the need of a center is less.

Neil Buesing (07:49):

I would say with Kafka, a lot of people are overwhelmed. They're overwhelmed, not only on the operational side, but how do I get my developers that do event-based driven design, to leverage this and be successful? So from a Kafka standpoint, it really has become more of a training mechanism, or an avenue to get success to their developers quicker. So I do say, I do see your point that the COE really, the focus of it has changed to being more of a training, learning, providing guidance to the development team.

Tim Berglund (08:29):

Sometimes advocacy within a larger organization if there's a technology change that an organization is trying to bring, and you've got lots of decentralized elements, the COE could even advocate. I don't know if anyone has ever said this, but somebody is about to, it's like a little embedded developer relations team, inside an organization oriented around a particular technology. Agree or disagree?

Neil Buesing (08:58):

Oh, I agree. I think that's a key point in it. I think those that were successful, even before Kafka doing it had exactly that in mind. It's about advocating for what makes sense for your organization and advocating that these tools provide the company value and it's important to them. And I think that's probably why you don't hear the term as much anymore, is it's really more about advocating for the developers so they can do good work at a client.

Tim Berglund (09:29):

Right, right. And I feel like I still hear it. I was reflecting on the fact that it's associated with a change boundary in time. That mature technologies, you don't have COEs for those. You always have them for new things where somebody's pushing for this thing to get adopted.

Neil Buesing (09:50):

Yeah.

Tim Berglund (09:51):

So I think you just gave us the why you do this, you're trying to bring value to a client company and you're toast if you don't do that. So that's important. You're trying to bring value to a client company. Even within that, it sounds like and this is definitely true. If you look at consulting companies, either on the boutique end, the giganomous end, or the in-between, which is where I see Object Partners. They have opinions, I think of like ThoughtWorks, is in terms of order of magnitude, I guess maybe a click bigger than you guys, but the same kind of thing, but lots of opinions there. You said you were on the Groovy and Grails bandwagon when that was hot, 10 years ago, as was I, quite early. And I think there's a definitive guide to Groovy is still visible in the background, when I shoot videos in my office. My head might be in the way of it, I'm not sure, maybe when I move, you can see it. It's blurry in any case, but if you recognize it's the spine, it's there.

Tim Berglund (10:55):

So yeah, I mean, consulting companies have opinions and Object Partners has opinions, and there are certain things within the broad mandate of wanting to deliver value and solve clients problems, there are ways that you do that. And solutions that you gravitate towards as an organization and even you as an individual, and event streaming is one of those. So it makes sense that you'd want to do that. You'd want to embed these organizations inside your client companies. Create them really. How do you do that, though? How do you make... I want to ask you, as a VP at a consulting company, how do you make a COE somewhere? And then after that, and I'll come back to this question, but I want to ask, what if you're a developer at one of those companies, how do you fit into that process? What if the direction of your career is you want to be on such a thing, how do you make it easy for Neil in your life to use you? But start with how you do it. How do you make a COE?

Neil Buesing (11:58):

Well, I can tell you how I've made a kind of and wrong decisions in the process-

Tim Berglund (12:05):

Always good. This is how I lead myself, so-

Neil Buesing (12:07):

Yeah. So I mean, as you said, the COE and what that means is rather broad. And it's changed over time. But one of the things that I've seen when it comes to getting Kafka into a client who wants to get other people to see it is you need to build a proof of value within that organization that shows how Kafka can modernize their systems. And usually, this proof of value, ideally, is done with team members within that organization. So you come up with, some agree to a time one month or two months if you're lucky. And you build out a team to take a slice of a technology burden that they have. Usually, this is I have data in a database that gets batch loaded to another system overnight and then I process it from there. And then I generate reports, and it's two days behind. How can I get that current? So you find whatever you can do in a month, and you show that there, and you show success in providing the value that Kafka can do it.

Neil Buesing (13:21):

Now in that process, this is where if you get like you mentioned about the developer and the company if you find people in the company that is enjoying Kafka or eventing, ideally, both in my case. It doesn't have to be Kafka, but that's what-

Tim Berglund (13:37):

But let's be honest.

Neil Buesing (13:38):

... my life is. So let's be honest. Yeah. You work through them to help build that. And you provide them the tooling to get there quickly. So that is at that point, you don't worry about Change Data Capture nuances on their end, you just get the data flowing into a topic, and you provide them those configurations to do it. One of the issues though, with doing a proof of value is you're going to run into roadblocks if you're trying to do that value against resources out of your control. So let's say I have data in an Oracle database that they want to do real-time eventing on. Well, getting a logging system enabled accessible and allowing you to get into a Kafka topic is going to be your four weeks. You're not going to get anything done. So you need to have your own Oracle database, ideally in a Docker container. And there are ways to do that, fill it up, get their data into there, and now you start showing them value without having to show how you integrate into their infrastructure. So that's key.

Neil Buesing (14:48):

Along with that, you need to help them learn how to write the software. So if you're writing streams apps to do something useful with Change Data Capture, you want to pair up with them and do it. And this is the hard part. And this is where things succeed or they fail. If they're available, and they're engaged, it's a win-win. You see their eyes light up with what you're able to do. And they see the value on it. If they're constantly pulled away type, dealing with other fires within the organization, they're not going to see that. But you can still build it for them and get that out there. But you're not going to get the buy-in from the developer. And that's the part where I say, I've seen where it failed, it's when you can't get that dedicated time from people within that organization, it gets tough.

Neil Buesing (15:41):

But again, you design this in a way that allows you to do it from beginning to end, without that developer present, without the DevOps people giving you integration. And then at the end of it, you show that use case that's delivered to them in a way that shows them the wow factor, that you were able to take a two-day process and show it in 30 seconds or five seconds. And I'm fortunate enough to be involved with two of those at a client right now. And seeing the management team be excited about how Kafka is going to solve or help them solve problems is very valuable.

Neil Buesing (16:21):

So yeah, do a proof of value, do it in a month, if you can. Get the team involved and along the way, build out those pieces that they could bring into a Center of Excellence. For example, people get very overwhelmed with how do they set up your producers and your consumers and your streams apps? Provide them a library to do that. And I've always said, "Don't wrap any client code." That's going to confuse them, they're not going to understand it. They're not going to embrace and learn Kafka, they're going to learn your wrapping of that client code. Let them do that, if and when they bring on Kafka longer term. You provide them a builder to say you want x off, for example, you want SSL certificates, you need to connect with Avro, and you want to use specific record, provide them a builder that gets them their clients quickly to that setting, but don't hide it from them. And I know, people do like to hide, and I've been one of those people, but you're not building an eventing platform, you're getting them to embrace one. So that's the key difference.

Tim Berglund (17:36):

One thing this podcast needs, Neil, that you're pointing out is more sound effects. And the sound effect right now would be some church organ thing indicating that the guest is preaching in a way that I like a lot. And I don't have that, here's the meta version of that sound effect that would be playing right now. And yeah, now in my experience, once you get to ten to the three developers in an organization, or near ten to the three, wrappers happen, and well, they just do. And I don't have to like it, but they are there. In fact, we had a recent, pretty interesting episode with Natan Silnitsky of Wix talking about really a Scala library. I mean, it was a wrapper, fully matured wrapper, its own Kafka library almost, that they built. But it was general-purpose enough that they've open sourced it. They're like, "Hey, Scala people, you want to use this to talk to Kafka."

Tim Berglund (18:42):

So it's not so much hiding as we wish the library was designed this way. So we built it. That said, everybody wraps, who's big? I guess there's a reason for it. But I'm with you, it... And to be fair, all of my engineering experience is in smaller teams. So maybe there's something we're both missing, but I feel you on that.

Neil Buesing (19:06):

Well, I think there's... I mean, part of it is, like you said when you get to the size of the organization and the purpose. And people who know me well, know certain frameworks that I avoid, and it's because they wrap and hide what I consider a very elegant library of Kafka. Why do I need to wrap that in someone else's opinion? But there are that and I try to be like you and get that. But if I'm building a Center of Excellence, and I'm building out something for people to learn, understand and embrace Kafka, then I'm pretty adamant about it, it's wrong, I need to do that, to wrap it. Because I'm not doing them any favors. I'm just giving them my opinions, which can, unfortunately, be wrong at some time. So I want them to get their own opinions.

Tim Berglund (19:53):

Right, right. And this also has to do with the whole discussion about leaky abstractions, when you try to build an abstraction around a Kafka library. Is that really the right place to be abstracting? And we all know it's going to leak and have... Are you able to design those leaks in good ways? That's a notoriously difficult API design task that I typically fail at. So-

Neil Buesing (20:22):

Agreed.

Tim Berglund (20:23):

... it's hard to do. Yes, you agree that I typically fail with that.

Neil Buesing (20:27):

I knew that's where you were going to go once I said it, but I agree that I do that too.

Tim Berglund (20:31):

Okay. Right. So back to you started with why Centers of Excellence can fail. And I think I heard that the primary reason be a lack of developer buy-in and centralized decision making, but lack of grassroots embraces. Did I get that right?

Neil Buesing (20:52):

You did. And that is very correct. Okay, if you're building a Center of Excellence, and you're listening to this, and like, I'm a company, and I'm wanting to build and gain Kafka within the organization, you have the buy-in, you're the person bringing that in. So maybe you don't need to find another person that has your passion for it. As a contractor, as a consultant coming into a client, they know I'm passionate about it, people can see that I love talking about this stuff, they are going to get that energy from me, but to be a successful consultant, I need to be able to leave or I'm not really doing them any good or any favors.

Neil Buesing (21:40):

So I need to get buy-in excitement within their own organization. And if I don't have that, they're going to on that transition period where you're trying to move on to the next shiny object, because you're a consultant and that's what you like to do, you're going to be saddened, disappointed in yourself, when you realize that no one's there to take the baton from you and continue driving this. And management brought you in to level up their team. Brought you in to make it so that they can let you go and when they can't let you go, or if they let you go and realize the project's going to go another direction you fail. And I've been at places where I feel like I didn't get that buy-in.

Neil Buesing (22:25):

And maybe the recognition would be, I don't have that buy-in, we need to course-correct now instead of trying to build out that one application that they wanted. While building out that one application isn't going to help them learn via Kafka. Now you brought in an application that requires Kafka, and no one wants to touch it because no one knows it. That doesn't do them any good. You might as well just use the more traditional technologies that they're familiar with.

Tim Berglund (22:54):

Absolutely. We've got, in the Confluent Developer Relations team, a thing called the developer journey. That's the five stages people go through. This is not really event streaming or Kafka related, but it's just the way we see technology adoption happening. Those stages are awaken, understand, play, build and advocate. Awaken where you become aware that there's a thing called Kafka, understand where maybe you watch the talk on it, you have a mental model of distributed logs and producers and consumers and serialization and something like that. Play is where you screw around with some example code and run something. You've spun up the Docker containers and produced the first 10 natural numbers to a topic or whatever. Build is where you actually start to make things in your domain. And that can be a big stage. That can be sketching together a POC, or deploying to production. Everything in there is in that build stage.

Tim Berglund (23:56):

But then the top one, you were just describing, the final stage is advocate. And not everybody gets there. But this is where you have gone through these previous stages, you've built something with the subject technology, in our case Kafka. And now you want to be an agent of change in your organization for more people to do what you've done. And you're saying that's the engine you're trying to build in the COE, is you're trying to build that team of advocates who have the experience and can help that technology adoption go on, even after the Object Partner's engagement is done.

Neil Buesing (24:33):

Correct. And I think your five steps is pretty cool in that you get into an engagement where they want you to do all five and they forget about getting other people even into that play stage, let alone the build stage.

Tim Berglund (24:48):

Yeah. Yeah, because understanding, that's fun for folks like us. We can go give talks and teach things and it feels good but hands have to get on keyboards, people then have to build things. And if you don't create advocates inside an organization, it won't have sustained growth.

Neil Buesing (25:08):

Correct.

Tim Berglund (25:10):

So what do you think of as being the deliverables of a Center of Excellence? When you've built this thing out, what do you leave with?

Neil Buesing (25:25):

That's an excellent question. Because I have notes in my table here, and I'm still trying to piece out what I consider what are deliverables. I can say some things that I know of, and let's start with that and I'm sure we can fill some in. The one thing that I've joked to many people and I have joked even to you is I always talk about Kafka on an airplane, which means I can run everything locally on my laptop, even if I'm not on a network, provided I have all my Docker images downloaded, which is you need to be on the network to do.

Neil Buesing (25:59):

But if I have that I can now play around, use Kafka without worrying about impacting anybody else within my organization. So if I want to do Change Data Capture from my sequel with the museum into a cluster, and then write it back out to Elastic Search, I can do that without bothering anyone. I think it's important for the Center of Excellence to provide an artifact to that team that allows them to do that play stage without asking for a Kafka cluster. Where do I write my topics to? And how do I play around with it? And there's a lot of tools already out there to do that. I mean, Confluent has the Command-line interface. You have the Docker images, I always start with the Docker images. And I tell people to make sure you have at least three brokers unless you're on a small machine. But ideally four.

Neil Buesing (26:51):

And the reason I say that is I want people to understand how Kafka works during this process. I don't want them just to build the application around it. And you need three brokers to make sure people understand replication, but you need the fourth, so they don't understand they're all exact replicas. When you have four brokers, people start to understand, "Oh, this is what's going on here. It's putting mixing up the data where it goes. Who's the leader, who's the follower. And this guy isn't even a follower for that partition." When you have three people they won't see that. So you want to build that out. And like I said, I use Kafka, or I use the Confluent Kafka containers, and I build out a rich ecosystem there.

Neil Buesing (27:33):

But then you need to start bringing in what pieces will that client be working with and add them to that environment, so Elastic, MySQL, Oracle, bring in ksqlDB if they're going to be using that. So get that tangible artifact of the Center of Excellence. The other one, again, I mentioned some client libraries, but as builders to help them write Java code in their frameworks. So if they're a Spring shop, give them a Spring Boot app, or give them some way to quickly initialize a producer in that ecosystem. Using Maven or Gradle, whatever they're used to using if you get to pick great, but if not pick what they're used to. And those again, are the tangible artifacts. The untangible, not that's a word but is really getting the buy-in from the other developers. And that's the hard one to layout on a piece of paper. I think you know it when you get it, and if you don't, if you can't find those people, then you need to talk to whoever brought you in to try to find that.

Neil Buesing (28:42):

From an operational side, that really varies from the organization on do they separate out development from ops? Or is it really one in the same? Are they going to use cloud provided services, so they don't need to manage that as much, but they still need to understand how to manage that? From there you want to bring them in monitoring. I think it's important for Center of Excellence to give people the assurance that what they built is running and running well. That's through Confluence control center, through Kibana dashboards, integrate to their data dog if they use it. Again, the whole point of this is to get them to feel less overwhelmed by bringing this in. By integrating it and making it as close to everything else that they're doing, but giving them the chance to learn and embrace it.

Tim Berglund (29:30):

I like that last part about helping them feel less overwhelmed. Because I try to, what's the word? Highlight this. I try to underscore this idea that event streaming is really new and really different. And building a completely reactive, completely event driven application. Patterns are different. The problems you're solving are the same, the tools you use to solve them really are different. And all the stuff that you're just going to pull out of your pocket for how you do that with building the system around a database, you need different things to pull out of your pocket. And that's unnerving to people who have a decade or two of experience. Senior level people who know how to do it, and now have to come up with this new way of doing it. And even less experienced people who know how to drive the legacy toolset, helping them feel comfortable, I think, is something we should be intentional about doing. And I'm glad that's one of the goals that you have for COEs, or-

Neil Buesing (30:39):

I think-

Tim Berglund (30:39):

... CsOE.

Neil Buesing (30:43):

I think the part when you said about understanding event streaming and where I get a lot of questions is, okay, what's Kafka going to respond to me when I publish? How do I know there's an error? And you go, you're not. And you just get the blank stare. Or you get the, it's not the blank stare, because we're all working remote and most people that I work with have their cameras off these days. But you get the silence. So the audio blank stare. And you mention, well, I can tell you, if I got it. I can tell you if I'm going to accept it. And I guess if you talk about rest, and you do the accept versus okay discussion people get it. But that event driven separation there is still a big one for... And it was big for me when I went through it too, for people that truly understand and trying to get them to understand that event streaming is very important.

Tim Berglund (31:47):

Yeah. Yeah, as I've said on the show before, you have to treat other services like adults. You wouldn't treat an adult that way. If you think of someone very close, like a spouse, you wouldn't say it's your turn to pick up the kids from school today. You don't call it 310 and make sure that they did it. Of course, even that is a little bit outdated thing. Everybody's doing school remote. But hey, maybe you're in a place where people are going to class. You just don't do that. That's not going to go over well. You say, "Okay, that's what you're doing. I trust you, you're an adult, you're going to get the job done." And you have to treat other services like that. So that's I find over and over again, a key message to communicate to people as they're becoming comfortable with event streaming is that they have to treat services like adults.

Neil Buesing (32:37):

Yeah, very much. And I think when you mentioned earlier, what specifics come out of a COE around Kafka, I think one of them is how do you handle errors and the concept of dead letter topics? And what does that mean to asynchronous responses? Do you have to monitor that? Is it manual? Building that up is going to take time and getting people to embrace it. So why not do that in what you're considering the team that's going to advocate for this technology, but making sure they understand that well?

Tim Berglund (33:09):

So building out patterns. I just try to avoid using the term best practices. I don't know why it just bugs me a little bit, but will just say it, best practices. Patterns and best practices inside the organization like when do you need a dead letter topic? And what do you do with it? How to think about a dead letter topic? Because again, that's a pattern that people used on an enterprise messaging system 10 years ago, or even more recently, and they were like, "Oh, yeah, dead letter queue. That's the thing. We need one all over the place." But bringing some nuanced and organization specific thinking about saying that, you're saying that's the task of the Center of Excellence. They ought to help form those opinions, help advocate those opinions, help teach people how to think about that topic in particular.

Neil Buesing (33:58):

Yep.

Tim Berglund (33:59):

Yeah. Cool. That's a good thing.

Neil Buesing (34:01):

Well, and then like, I tell people, "We'll just write it to a dead letter topic and let people listen for events on it and handle it accordingly." And they're like, "Well, how do they get emails?" You don't. So then eventually, "Well, okay, we're going to have to build a service that potentially listens to that topic and sends out a different means of communication." So then you understand what are the infrastructure pieces that a company may need because they're not ready to let all these other individual pieces within their organization do that heavy lifting.

Tim Berglund (34:38):

Right. In fact, that's always my guidance on dead letter topics is if there isn't something listening that's sending an email, or building a web UI, and there's a human being whose job it is to look at the malformed document and adjust the business process through manual exception handling, if you're not doing all that stuff, you may not need a dead letter topic, but that would be an example of an opinion, which, number one could be controversial and ought to be adjudicated per organization by the Center of Excellence or some similar group.

Neil Buesing (35:18):

Yeah.

Tim Berglund (35:18):

You want to keep policies light and not make this a bunch of laws people have to follow. But when I'm new to Kafka, I don't know that. And I don't have to worry about that. So it's good for somebody to tell me that.

Neil Buesing (35:28):

Yeah.

Tim Berglund (35:29):

If I'm a developer at an organization and I am interested in advocating for Kafka, okay? So that's like you or me, but we're employed somewhere in some company, whose business is not consulting and not technology products, but some other thing. And we want to see there be more Kafka, how do I cooperate with a Neil? How do I make myself a good candidate for a Kafka Center of Excellence for the Neil Buesing in my life and if I don't have a Neil Buesing in my life, what do I do? Is there a way I can lead an effort like this myself?

Neil Buesing (36:12):

Let me start with the second because I'm trying to still figure out the best way to answer the first so I'm stalling-

Tim Berglund (36:19):

Cool.

Neil Buesing (36:19):

... a little bit.

Tim Berglund (36:20):

That's how we do it.

Neil Buesing (36:22):

If you're looking to level set yourself up, and also start this within your organization, I mean, there's a lot of resources that Confluent offers through your white papers and your blog. And I always tell people to build up your own reference material that seems relevant to your organization. So what I do for Object Partners is I have a document of all the books I recommend, all the settings I recommend backed by other articles. I try to make it look like I'm not the expert, but I'm providing them a document that lets them quickly get to the expert material and summarize it for them.

Neil Buesing (37:08):

So I become a news aggregator for them around all things Kafka. So when I'm, for example, I need to let people know about exactly one semantics, I usually refer to them too, oh, I forget the... Strange Loop Talk by Jason-

Tim Berglund (37:26):

Yes.

Neil Buesing (37:27):

... back in 2017.

Tim Berglund (37:28):

Jason Gustafson.

Neil Buesing (37:29):

Yep, that's my go-to material.

Tim Berglund (37:30):

Kind of the definitive guide right there.

Neil Buesing (37:32):

Exactly. And it's in video form, which it usually is, I say what some people like. And it's harder to find than necessarily the blog forum. So I like to point out to them. Consumer group rebalancing, I think that's a Strange Loop by Gwen a year later. So build that document up.

Tim Berglund (37:54):

That's how I learned, consumer rebalancing, by the way.

Neil Buesing (37:56):

I think that's how most people who learn it well have learned it, is by watching that video. I definitely learned... I realized how dumb the brokers are by watching and re-watching that video.

Tim Berglund (38:09):

Yep.

Neil Buesing (38:09):

I knew the brokers were dumb. And there's so much on the client.

Tim Berglund (38:13):

But now you were like, "Oh, that's all on the client? I thought it was-"

Neil Buesing (38:15):

Yeah.

Tim Berglund (38:15):

Right?

Neil Buesing (38:16):

Yeah.

Tim Berglund (38:16):

Me too. Yeah.

Neil Buesing (38:17):

And it's pretty amazing. It's pretty cool. To build up that... And again, if you're an expert in Kafka, or you don't know anything about Kafka that you're on the bandwagon, build up a set of material that isn't using you as the basis of what they should learn. But just again, collecting that material. You can summarize it. So one of the things that I do is people are overwhelmed by what their client setting should be. And I use Confluence white paper on tuning, optimizing your Apache Kafka deployment. And I build an Excel spreadsheet, or I guess, a Google Sheet of all the settings on one page that shows them how they relate. But I can always point back to that and say, "This is why I'm telling you this." Because I don't want people to say, "Well, I set X to negative one, because Neil told me." No, I want you to set X to R because this document told you to and it's backed by more research, more understanding than just myself. I happen to agree with it, so I'm going to share it.

Neil Buesing (39:25):

So if you're the person that like I said, regardless of your expertise in Kafka using other documentation is a key way to get others to buy into what you're advocating for. And I guess to your earlier question then, if you have someone that's available to you that knows this then I guess use them to do that for you. Or use them to help build that for you with them. So you can say, "This doesn't make sense." Or, "We're a Golang shop. So should we use liberty-kafka or should we use the Go Native client and why?" Or you have someone that says, "I'm both a Go and Java shop." It's like, "Well, then point them to the partition algorithm of liberty-kafka and remind them that that's not the same by default as Java and stick that in that Center of Excellent document." Because that's the one thing that's going to keep someone up at 3:0 AM because their partitions aren't aligning because they didn't know that existed. So those little nuggets-

Tim Berglund (40:28):

Because they have different clients. Yes, yes.

Neil Buesing (40:28):

Yeah. Those little nuggets can A, give people respect for what you're trying to provide them. And it's usually the thing that will save them with a gnarly bug at like I said, 3:00 AM.

Tim Berglund (40:43):

What keeps me up at 3:00 AM is just insomnia. But that-

Neil Buesing (40:46):

Well, that's, yeah.

Tim Berglund (40:47):

... will get other people. And you said the Golang, use the Native driver or the liberty-kafka base driver. So it's the end of September 2020, as Neil and I are recording this. This will be released a few weeks from now in the future. But there's been a Twitter meme in the last couple of weeks of a video of a street fight. It looks like some vaguely European looking narrow city with a bunch of alfresco dining, lightweight cafe chairs outside, and the guys are throwing the chairs back and forth at each other. Have you seen this?

Neil Buesing (41:24):

I have not.

Tim Berglund (41:25):

Okay, well, I'm going to need to find-

Neil Buesing (41:28):

[crosstalk 00:41:28].

Tim Berglund (41:28):

... that instance of it. Right, yeah. But you'll ask some seemingly anodyne question that happens to be radioactive within a certain community and then this happened. Well, should I use the Native Golang driver, or the liberty-kafka one is definitely the-

Neil Buesing (41:46):

Yeah.

Tim Berglund (41:47):

Yeah. Then you've got the guys throwing chairs at each other up and down the street.

Neil Buesing (41:51):

So I don't think I've seen the original meme that started off. I think I've seen enough of the others to make me wonder where that all started, but not curious enough to trace back the roots. So I'll have to do that.

Tim Berglund (42:02):

Got it. Yeah, no, we'll need to find a copy of that video and tweet. Hey, you should use the liberty-kafka Go driver. And then this happened. Anyway, Go jokes. Wrapping up, how about metrics? Funny question coming from me, because I'm never the metrics guy. But I imagine you get asked to do this. How does one measure the effectiveness of a COE?

Neil Buesing (42:29):

Metrics as in adoptance and [crosstalk 00:42:34].

Tim Berglund (42:34):

Yeah. Is there anything next typically, that you recommend? I mean, and no, we don't do that is actually a great answer. But I'm just curious.

Neil Buesing (42:43):

Yeah. Any metrics around that is not been established or fully collected. I guess I have, in my mind, what I see as success or not so successful based on, how many different projects or different organizations are starting to use Kafka? Not only the potential of the enterprise cluster that maybe you help stood up but other ones. So the metrics really is the number of developers that are asking you questions and learning about it. And the organizations that they're part of.

Neil Buesing (43:24):

So I like to keep track of how many people are asking me Kafka from which groups so I can tell, "Oh, the news is spreading on that." Or there's excitement for it. Then I was at another client, where there's none of that. So it's like, "Well, that's a zero basically. I'm only getting asked by the people that are forced to take over the project or are asked to work on the project." So then that gets a little deflating, but we don't have success metrics on there. I'm trying to figure out what a good one would be. I think it's all a heuristic based feelings from very scientific of me. I know, feelings and having discussions with others.

Tim Berglund (44:10):

Honestly, the intuition of experts is useful. I mean, we use measurements. Measurements have an important role in a lot of things. But in particular, when you're needing to validate your intuition against the cognitive biases that can potentially sway it, measurements are good, but absent that, honestly, the intuition of an experienced person is a very powerful filter. So I think that's a great answer.

Neil Buesing (44:42):

I try.

Tim Berglund (44:44):

My guest today has been Neil Buesing. Neil, thanks for being a part of Streaming Audio.

Neil Buesing (44:48):

Thanks, Tim. Appreciate it.

Tim Berglund (44:49):

Hey, you know what you get for listening to the end, some free Confluent Cloud. Use the promo code 60PDCAST. That's 6-0-P-D-C-A-S-T to get an additional $60 of free Confluent Cloud usage. Be sure to activate it by December 31st, 2021 and use it within 90 days after activation. And any unused promo value on the expiration date will be forfeit and there are limited number of codes available, so don't miss out. Anyway, as always, I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can always reach out to me @tlberglund on Twitter. That's T-L-B E-R-G-L-U-N-D. Or you can leave a comment on a YouTube video or reach out in our community Slack. There's a Slack sign-up link in the show notes if you'd like to join. And while you're at it, please subscribe to our YouTube channel and to this podcast, where ever fine podcasts are sold. And if you subscribed through Apple podcasts, be sure to leave us a review there. That helps other people discover us, which we think is a good thing. So, thanks for your support and we'll see you next time.