Domain-Driven Design and Apache Kafka with Paul Rayner

Tim Berglund (00:00):

Are you a domain-driven design practitioner? If not, do you often find yourself bumping into DDD people in the Kafka community? I do. Today I interviewed Paul Rayner, CEO of software design consulting firm Virtual Genius about domain-driven design and its relationship to event streaming. Check it out on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud.

Tim Berglund (00:30):

Hello and welcome back to another episode of Streaming Audio. I'm your host Tim Berglund and I'm joined in the virtual studio today by Paul Rayner who—it's funny that I say virtual studio. I know I said that every time, but Paul's the CEO of a company called Virtual Genius, a software design consulting firm, founder and chair of explore DDD, a domain-driven design conference in Denver, author of the EventStorming handbook and other things. And he's also a dear friend of mine. So Paul's nice to have you on the show.

Paul Rayner (00:59):

Yeah, glad to be here.

Tim Berglund (01:01):

How did you? So I've kind of given you up as the domain-driven design guy and that is certainly how I think of you and that is a specialty of yours. And I'm going to give you an opportunity to explain, uh, what that is and we're going to unpack what it has to do with Kafka, which it seems to be a lot based on what I see in the world. But, uh, before we get there, um, how I always like to kind of ask people about their professional journey. How did you get to where you are?

Paul Rayner (01:29):

Okay. I started out in the 90s out of college doing C and Fortran programming for mine planning and mine modeling software. And so my early experience of software development was occasionally getting on a 10CT aircraft and flying for two hours out into the middle of the Australian Outback to a coal mine or a gold mine. And sitting down with a geologist or a geotechnical engineer and figuring out what they did and how they used our software and then figuring out what they wanted it to do and coding that up.

Paul Rayner (02:05):

And then, you know, this is the 90s. So agility, continuous delivery in those days was putting the software, a copy of the software on a dat and sending it to them. And they would get it a couple of days later and then load it up and start working around with it and load it up. So the special-purpose computer, some embedded setting or, yeah, this was the 90s. So these were Silicon graphics workstations, which were pretty high-end machines at the time. And I still, yeah, I still have fun feelings for the SGI Indigos and Crimsons so yeah, they would load it up.

Paul Rayner (02:43):

And it was, uh, my, so my early experience was working in these languages and sitting down with people that were experts in their field, reading some of their books and, and trying figure out how to represent what they were doing in three dimensions on the screen and provide tools for them to, you know, model all bodies and design roads and things like aquifer modeling, geotechnical modeling—those types of things that are useful in that kind of context.

Paul Rayner (03:19):

And then along came Java in the late 90s and did that for a while and then .NET and then started moving into, um, Ruby probably about, I don't know, 2000 I can't even…

Tim Berglund (03:41):

Seven? Yeah, I'm getting parts of Paul's story or form. Sorry. Yeah, we had that lunch at that Thai place once when I had done some playing around with Rails when it was the thing that every Java program was playing around with. Right. Yeah, let's try it. I think I gave you the world's worst Ruby on Rails tutorial and you sort of took the ball around with it and actually got good at it.

Paul Rayner (04:00):

Right. And more on the Ruby side than the Rail side. I think cause I was always more attracted to the idea of what I can do with the actual language using, you know, plain old Ruby objects than the Rails framework itself. Even though a lot of the companies that I've consulted with over the years, extensively Rails shops, and it was in terms of getting into DDD. That was probably about two around that same kind of time frame.

Paul Rayner (04:29):

Actually I was working at a company and we were doing extended warranty software and we're really struggling with that. And one of the things that really helped was reading Eric Evans’ book on domain-driven design, which I was familiar with the classic gang of four books and um, Martin Fowler's book on refactoring and Joshua Kerievsky on refactoring to patents. And those had all been really helpful and you know, I read my Fisher books on test-driven development and behavior-driven development.

Paul Rayner (05:00):

But DDD really kind of opened my eyes to the idea that you could actually take that complexity that you are struggling within the business and produce a model in the software that could help get that under control and, and also do it in collaboration with the people you're working with. So with the people. So it's kind of hearkening back to my early experiences as a developer, where, you know, sitting down with someone that's an expert in geotechnical engineering and figuring out how to build tools to help them. It was a, it felt very similar to that in a lot of ways when I came across DDD for the first time yet.

Tim Berglund (05:37):

Some differences there—I think we have everybody just thinks, uh, mining, geotechnical engineering, you know, we'd be, there's like some geophysics involved there. Um, there is, uh, a lot of really interesting mathematics that…that goes into some of that stuff.

Tim Berglund (05:58):

And all of the systems underlying the software you're written, you're writing are...

Tim Berglund (06:04):

And I'm making some assumptions here and if I'm wrong, just let me run with it. Um, but their physical systems and the models are mathematical models and they're hard. They're very hard. And maybe sometimes there's some business sort of domains mixed in there too. But when you're writing software to model, to, to model, um, represent a mathematical model, it might be difficult because the model is complex and it's math that you don't know and you have to learn. And the process of learning that math is hard, but the programming language that you're using is a, it's a formal language, right?

Tim Berglund (06:40):

It's this machine that you're configuring to do a thing. And the mathematical model underneath that machine is also mechanistic. So the mapping of, you know, in the case of like strange geophysical models are at that same time I was writing firmware to control, uh, embedded modems and you know, telecom systems and stuff like that. When it’s a mathematical model underneath the mathematical, make it a mechanistic processor executing your code. It might be hard, but at least they're kind of like in the same world.

Tim Berglund (07:10):

And when you get into the extended warranty stuff, everybody's gut reaction is, well that geotechnical mining stuff sounds really hard. I bet the math is hard. Warranties are dumb. That's easy. I could do that. Right? But the reality is it’s so not true. Right?

Tim Berglund (07:25):

Exactly. And I mean you're right with mathematical models and what at the time was called computational geometry. There's, you know, you're trying to do something like represent an all body as some kind of block model. And we, we at the time , developed our own what would be called now and no SQL store to represent that. Or you're taking data points and triangulating them to build some kind of 2D surface. But when you're talking about modeling, some kind of business process, well then, you have to deal with just all the complexities that go along with different roles in that process with handling how things go wrong and then how do you represent certain concepts?

Paul Rayner (08:06):

So even basic things like naming is hard, right? So is it a warranty? Is it a contract, is it a customer agreement? What do we call this thing? If we're, if we're building something new and, and then factor into that, there's probably an existing system that calls it something else that you have to integrate with somehow and figuring out how those integrations are going to work and all of that. You're dealing with invisible things and sometimes those invisible things don't even have names yet.

Paul Rayner (08:41):

So you're trying to deal with all of that. And so it is hard. It's a nontrivial thing. And when you have, you know, I've been working with teams that are modeling a lifetime of financial transactions, trying to figure out how to deploy people out to do, uh, in field assessments and all kinds of things like that. Then sometimes, particularly if you start up, you have to figure all that stuff out often from scratch, how are you going to do it?

Tim Berglund (09:13):

Right. And it's a little early in the conversation to uh, come to conclusion as philosophically subtle is. I think where I'm coming from right now, but I don't care. I think what is happening there is, uh, go for it. Go for it. I'm doing it. I'm doing it. Like if, if there's math and you're writing software to do math, well the, the machine that you're configuring the computer sorta does the same kind of thing as math. It implements step-by-step processes to achieve a result.

Tim Berglund (09:46):

And a company that provides extended warranties is a bunch of people working on a business process. And that's not fundamentally mechanical, that's fundamentally mental and it's the cooperation of a bunch of minds doing a thing. And some of the things you, you just, uh, you just hinted at there like what happens when something goes wrong? Well, there's all kinds of exception management that people can learn to do and do effortlessly when, Oh wait, that feels not filled out well. I can just fill it in cause I see this other place or Oh that feels not filled out.

Tim Berglund (10:19):

Uh, then I know this needs to happen. Or you know, those validation errors people learn without thinking to handle a giant set of those. You said you're dealing with things that are invisible. There are all these quote-unquote entities out in the, in, you know, in the business environment and people can traffic in those entities and not have names for them, not think of them as things but still get the work done. And once they get good at their job, they might even be able to get it done while they're listening to music or talking or whatever.

Tim Berglund (10:49):

You know, like, like just kind of popping through life. But when you try to make that mechanical, suddenly it's the hardest thing in the world.

Paul Rayner (10:56):

Right. Because what we really sort of talking about here is the idea of taking these concepts and modeling them in the software. Not in a mechanistic way necessarily, but having something like if you're talking about Java or C# or some other kind of object-oriented software, then having something that represents those entities in the software as a class or an object in Ruby or it could be an actor in Erlang or Elixir or some other more functional language and taking these implicit concepts and making them explicit in some kind of software model with these concepts can interact with each other.

Paul Rayner (11:38):

And it's not about trying to come up with a realistic model in the sense that you're trying to maybe represent that process the way you would represent an all body but providing the concepts in the software in such a way they're actually useful, right? You're developing a model that as a tool is useful. So in unextended warranties, you want a model that can or models, um, multiple models that can serve people in the call center.

Paul Rayner (12:08):

People that are loading the warranty information that's coming in from the companies that uh, such as, you know, best buy or whatever the companies are that are actually selling the warranties. At the point of sale so that warranty data is coming in and then you have to manage the claims process if there are repairs or replacements happening on dishwashers and refrigerators and TVs and things like that. So there's multiple roles involved in all of that. There's dealing with things like tracking the financials on all the financial transactions and, and how do you handle that and, and recognizing that one model isn't going to pull all that off, right?

Paul Rayner (12:48):

You have to have different ways of conceiving of these things and boundaries between those different ways of conceiving of things so that you can actually pull off something that complex. Hmm.

Tim Burgland (12:58): So that you're helping me tiptoe away from my argument for a, uh, substance dualistic theory of mind from business software. Uh, that argument is in there. It's lurking in there. We'll come back to that on a few. Maybe you could join me on a future episode. We can have that discussion, but you were out before the day hosts, right. But necessary, absolutely necessary upon...and I think instead you were leading us back in a potentially more healthy direction towards telling us what domain-driven design is. And you were kind of describing some key concepts there.

Tim Burgland (13:35): So if you're to take it from the top, imagine we have listeners who don't participate in this world. You discovered it, extended warranties that you needed it. What is domain-driven design?

Paul Rayner (13:47):

Okay, well it's, it's kinda like a, for a team or an organization that's struggling with a complex process or complex rules, it's a way of managing that through software effectively. So the idea is that you take that business complexity and you develop models in the software that can simplify that complexity down and actually do useful things. And you do that in a collaborative way. So you're having the people that are the business experts, um, collaborate with the technical people, with the developers, testers, UX, all of that to come up with a shared language for describing the kinds of problems they're trying to solve and then representing that in the software.

Paul Rayner (14:34):

And to pull that off because that's not easy. You have to have some kind of iterative process where you're iterating over the design, you're iterating over the code and improving things. So refactoring the code and you're doing it in an incremental kind of way. So you know, the, you know, the shorthand for that is usually the idea of an agile process where you have iterative and incremental development happening and a team that's able to do that.

Paul Rayner (15:05):

And there's some other stuff in DDD related to more strategic design concerns, um, around how, how and where do you define boundaries between modules and between these different models and how do you integrate them and the kinds of relationships between those. That's part of the strategic design aspect. And then there's a bunch of tooling in DDD around also making sure you're focusing your design effort where it really matters. And um, you mentioned event storming earlier. So that's a tool, that's a technique that's developed in the last few years for being able to do a better job of that collaborative piece that I was talking about, especially around business processes.

Paul Rayner (15:46):

So, so it's a set of principles and evolving practices around dealing with complexity in business domains is probably the short elevator pitch for it. So it's not going to work, you know, it's not going to be the right tool for a team that's working on a pretty straightforward kind of problem. Um, but for teams that, uh, dealing with a fair bit of complexity, then it's a set of tools and techniques for approaching that in a way that's going to help them be actually able to get stuff done quickly and then accelerate, hopefully accelerate that development over time as they continue to improve the model and the language.

Tim Bergland (16:30): And, uh, it seems to come particularly from a complex business domain. So what we think of as business software, is that right?

Paul Rayner (16:38):

Right. Which, you know, it's funny because anytime someone has said to me, ah this isn't a very complex domain and then we start talking about it and getting into the details. I say actually this is way more complex than I was led to believe.

Tim Berglund (16:52):

So that's either somebody who hasn't engaged the domain yet or somebody who is so mentally immersed in it that their mental processes are effortlessly reasoning about the domain and they've never tried to write code with it or write, write code and represent it.

Paul Rayner (17:06):

Right. And sometimes something that, you know, one person's simple domain like, I mean, you know, if I want to send an email then I could use, you know, some third party API to send an email. Or if I want to process a payment, I could use Stripe to do that. And from the, from my perspective, processing a payment, uh, through Stripe is very straightforward. And the reason for that is because the people I assume that have developed that API and that huge iceberg, what I'm assuming is a huge, huge iceberg of, um, ah, third party gateways and processing that they have to do to actually make those payments happen is not simple.

Tim Berglund (17:46):

Every species of exception management possible and right. You know, so what from the outside for me is a supporting what DDD would call a generic or a supporting domain where it's, it's something that is pretty straightforward for me and generic to the people actually developing that software as their, their core domain, like their differentiator. There's an incredible amount of complexity that they have to deal with. So I think it varies on what you're trying to do in that, in that particular domain, how you're trying to use it as to where the complexity lives.

Paul Rayner (18:23):

Agreed. Now I liked your, uh, your elevator pitch summary, um, that it's a set of tools and practices for helping manage complexity and software designers. Something like that. I didn't write the words down. We can rewind later, but that's good for close enough yet. Um, for a fast elevator or not many floors. Um, but for like a consulting engagement of yours where there's a team that you're trying to teach them things. Um, and, and if I could keep the question ignorant, you know, they're saying, teach us how to be good at designing software.

Tim Berglund (18:56):

Um, and let's say they're, they already have some sort of pro forma agile process in place where they're doing sprints and, and you know, demos at the end of sprints and retrospectives and, and they're not horrible at that. Uh, what's your outline? What are you—

Paul Rayner (19:10):

Teaching people? Yeah, so when I teach DDD, it's, it's along the line of those three different areas I mentioned. So one is how to get better at collaborating and doing collaborative modeling. And event storming is, I think, a killer tool in the tool belt that they can add to whatever other tools they're already using. Right? So maybe they're doing user story mapping, maybe they have a UX person that's really good at, um, you know, the interaction design and, and the kind of value they bring.

Paul Rayner (19:42):

So there's the collaborative modeling piece. There's the strategic design side of things to make sure that they understand that they are working on things that, you know, they're putting the design effort where it really, really matters. Um, and some, and, and, and then related to that is identifying and enforcing boundaries in the software is really, really important. Um, at a strategic side, which, and, and related to that is how teams align with those boundaries.

Paul Rayner (20:16):

Um, and then the third part is modeling and code. So using some of the patterns that DDD presents for identifying what's an entity, what's a value, what's an event, um, what's the role of domain services versus application services. That's where we also look at things like hexagonal architecture and messaging patents and, and those types of things as well. So that it connects to, you know, the, the, the language they're trying to develop, which DDD would call a ubiquitous language is expressed in the code through.

Paul Rayner (20:55):

Um, so you have an S—what Eric woud (Eric Evans), would call a supple design in the code. So a design that can evolve and improve over time and provide you with a vocabulary for building up a more sophisticated model.

Tim Berglund (21:11):

Not ubiquitous language is a set of words to describe things in the business. And everybody agrees both in the code. And uh, you know, the business stakeholders who, who don't touch the code, they all agree to use those same words for things like they call it a warranty. And not a contract. And various kinds of exceptions have particular names.

Paul Rayner (21:33):

Exactly. And you know, this was a simple reason for that is when you get down to the code, you have to call it something. And if you're not consistent with the naming of things in the code, then you know, you're on the fast track to spaghetti code because somebody's coming in and looking at the code, it's like, well, I see one trip contract and I see warranty and which is, are they the same thing? Are they a different thing? I don't know. So having consistency that way and it goes beyond just simple naming.

Paul Rayner (22:03):

It's also thinking about the life cycle of things, the events that can happen over time to an entity. It also relates to how those entities interact with each other. Um, and then beyond that, recognizing that you're not trying to come up with one language for the entire organization, but it applies within a particular context and understanding what that context is. So an example would be, um, you know, when you're talking about managing claims for insurance, a claim is a very rich concept in, in that particular context, but, but to the accounting department to accounts payable, it's really just money going out the door.

Paul Rayner (22:49):

I mean, it's not just that, but it's, it's a much more narrow concept of what a claim is and it's in the context of financials rather than in the context of managing repairs and dealing with perhaps a service or management company and all of those kinds of things. So it's recognizing that different, like that language is not one language forged in the fires of Mordor to rule the entire organization, but it's something that is within the context of a particular system.

Tim Berglund (23:23):

Okay. When you put it that way, I kind of wish that it were that one language forged in the fires of Mordor and can only be destroyed by being cast back into the fires once it came. Uh, once they came, no, it didn't actually, it didn't that, that whole, the ring thing scheme didn't go well. Uh, finally. So are you, um, is, is that...

Tim Berglund (23:47):

That context in which the language is applicable? Uh, I, I know, you know, we're about 20, 25 minutes into our conversation. Haven't said bounded context yet and anyone playing the domain-driven design drinking game is uh, disappointed I think also still quite sober. Right? That's intentional. Yup. Yes. I figured you were doing that. Is that language you try to avoid? It sounds like you're talking about it, but you're not saying it. Could you, and I know anybody who thinks about DDD knows or has heard that heard that term so, right.

Paul Rayner (24:19):

So, part of the reason why it's tricky is because people immediately jumped to, well what does that look like in terms of the implementation? But it's a really, I think it's a fairly simple, straightforward idea. Um, at least it is to me. So if, if you think about, so you've got an application, some kind of application, then the idea is it expresses some kind of model, some kind of model of the domain. And basically that means if whenever you see a particular name of something, it should mean the same, like the semantics for concepts when it, wherever you look in that, um, that code should mean the same thing.

Paul Rayner (25:02):

And so bounded context is just a way of saying we need to have a boundary around a particular domain model and know where it starts and know where it ends. We need to know where the boundary is and it's a semantic thing that is then instantiated through things like modules or namespaces or uh, just an application, a standalone application or a microservice could be, I was hoping that word, that's the rest of the, it doesn't have to be right. Let's say you have three microservices that are all collaborating somehow around the life cycle of an extended warranty and all use the same names and meanings for those terms.

Paul Rayner (25:44):

Then they've, they're much more likely to express a single bounded context than three individual ones. Whereas if you have a single microservice that is talking about extended warranties and every other microservice in your ecosystem is talking about contracts, then you know you've got at least two bounded contexts. So what I'm getting at is it's a semantic thing that is then instantiated in the code. And the concern in DDD is to have good boundaries around those so that you know where one model starts and another model ends and, and it means you have to pay attention to integration between them as well.

Paul Rayner (26:21):

Because if you're publishing out events from a particular microservice that you know, those events have a certain semantic payload that is going with them in terms of what they name things. And also the data that is contained in those things is expressing a certain meaning and a certain type that is going to be exported out to any collaborating service that is then either going to be conformed to or transforms, translated into something else.

Paul Rayner (26:53):

So DDD is very much concerned with the name, you know, the, the semantics of that and, and how that plays out, which means that, uh, you know, regardless of whether you're using Kafka or, um, you know, whatever the underlying technology is, the principles still apply.

Tim Berglund (27:15):

Yes. Um, and it, it does sound like you just allowed for, um, contexts to share schema. Like I, I can, I can exchange objects between contexts and that that can be part of my contract.

Paul Rayner (27m 29s):

Is that right? Right. Um, and so when we were talking about, um, strategic design in DDD, this, the notion of mapping out the various contexts that are at play and also the relationships between them. So what you just described is most likely to be a conformist relationship where one context is conforming to the model of the other one. So if you've got like microservice A and B and A is emitting out events in its own language and B is just basically conforming to those events, then B is conformist to A.

Paul Rayner (28:07):

Whereas if this kind, yeah, if this some kind of translation layer that the B has that says, okay, well if I get a message from A, I'm going to translate it into my own language B, well then you know, it's some kind of anticorruption layer then that's protecting context B from this upstream context A and it's so, it's…

Tim Berglund (28:32):

It seems quite attentive to the historical problems of schema evolution and their overall terribleness.

Paul Rayner (28:38):

Right, right. Yeah. And, and paying attention to that, right, because I mean, I, it seems to me that data is not mutual. It brings assumptions with it about behavior and context. And so, you know, if somebody is just naively consuming data from another context and they're also going to have to deal with what does that mean in terms of where that um, like if it's expressing, if the events that it's receiving are expressing something about the life cycle of an entity in another context, then there were certain assumptions about what that life cycle means and what things are possible and what things aren't possible and, and you know, DDD cares about that kind of stuff. It says you have to pay attention to that or you could be in for a world of hurt.

Tim Berglund (29:29):

Yeah. Yeah. Which you will. And the, and like everybody knows that anybody who's, who's built, um, who's ever had two things talk to each other and then maintained those two things for, for longer than a few days, knows that the language that they used to talk to each other or rather just the, you know, the schema of the objects changes and uh, that's always hard ever-evolving schema in, in database is proper, is hard.

Tim Berglund (29:57):

Doing it in Kafka is hard. We've got a whole, you know, there's the whole confluence schema registry and uh, tools and, and well, really practices that people have built up around it that attend to that same problem. So it's nice that there's some, uh, sort of procedural support here in this set of tools and techniques.

Paul Rayner (30:17):

Right, yeah Cause DDD is trying to get you to think about those things as first-class ideas rather than things that just come.

Tim Berglund (30:28):

Rather than we're sure you got your schema right? It might change at some point accidentally. And so you deal with it, it's more saying, Oh well no, this, this is going to evolve all the time and you know, who's in charge and how do you keep it from breaking you.

Paul Rayner (30:42):

And a really fronting schema expresses some kind of language that hopefully is coherent and consistent and is evolving over time and you have to manage that. So there's a set of ways of thinking about that and approaching those kinds of areas. And like I said, it's still an evolving thing. It's not like DDD has all the answers for these, but you know, these are the kinds of things that people that would be attending a DDD conference or you know, reading books on DDD would be caring about, right. Is how to apply these kinds of things. And you know.

Tim Berglund (31:15):

Attending an DDD conference link in the show notes to explore DDD folks. Uh, it's in September. Yes. I think, uh, just the week after labor day, like 13th and 15th?

Paul Rayner (31:31):

You, yeah. You actually said it's in Denver and it has been the last three years, but this year we are actually literally leveling up. We're taking it up to Keystone resort in Colorado and it'll be September 14th through 18th.

Paul Rayner (31:43):

So, um, explore the DVD, Keystone Club. It's actually going to be at Keystone Resort. So I keep telling people, bring, bring the family, you know.

Tim Berglund (31:54):

Oh my goodness. Yes. So, uh, September at, uh, what's that like 9,000 feet, 8,000 feet, something like that. Nights will be chilly. Days will probably be, you never know, days are probably gonna be sunny and delightful and they'll still be flowers and just kind of mountain everything. Amazing.

Paul Rayner (32:12):

Last year we went up, last year we went up off to the conference, I think it was the week after and the full colors were kind of, you know, looking really nice. So I picked that week in September because we're trying to make sure that, you know, the weather is as nice as we can hopefully get. And the conference is designed for lots of collab. You know, the idea is to get people bumping up against each other in the same space.

Paul Rayner (32:43):

And if people go to a talk and they really enjoy it and they want to talk about it, then go on a hike and you know, dig into it. So we're trying to really optimize for there those hallway conversations except don't have them in hallways, you know, have them walking around the resort walking around Keystone or you know, enjoying

Tim Berglund (33:05):

Anywhere in the majestic Rocky Mountains.

Tim Berglund (33:09):

Exactly. Uh, and I'm not here to plug the conference. It is, it is a good thing to be aware of. Um, if you're interested in this and there is likely to be some Kafka content there and we're, we're, we're getting to is there is a heavy correlation between what I'm going to call interesting Kafka use, right? That's not Kafka as a big pipe, but Kafka as an architecturally interesting element in ways that you wrote me, talk about on this podcast.

Tim Berglund (33:33):

If you're a longtime listener and domain-driven design, those things come together. And what's funny is that I talk about those architectural concepts a lot and I routinely have people in my audiences who are using domain-driven design languages like language. I mean they, they knew who Eric Evans is, they know who you are. Um, but, and, and the, the funny thing like, you know, listeners think of you and your friends who are software developers who you've never directly worked with.

Tim Berglund (34:03):

What do you talk about? And apparently Paul and I never talk about domain-driven design cause I know like Wikipedia, the first two pages, four paragraphs of Wikipedia, content level, domain-driven design. And I'm kind of learning some things here today for my friends. This is good. Thank you, Paul. I think we use our time wisely together, but we just never talk about this. And event storming. Oh, go ahead.

Paul Rayner (34:29):

Well, it's just, I say that it's certainly been true over the last three years that um, a number of people, like a good percentage of people attending explored DDD are using Kafka and related tools for event streaming and, and you know, dealing with asynchronous messaging and, and building those into their architectures. So, um, I see a lot of that and you know, a number of my clients have used Kafka for those types of solving those kinds of problems as well.

Paul Rayner (34:60):

So it's something I keep bumping up against in not, um, and, and so, you know, a lot of what I'm doing is looking at things from an a, an event-based perspective that's certainly been a big shift in the DDD community in the last five to 10 years is thinking about things, thinking about life cycle of entities, more from the perspective of a flow of events over time, expressing state changes rather than a static view of state that we really don't know how we got here to this state kind of thing.

Paul (35:38): So that's become something that has become much more prevalent. So, Oh, you were asking about events. Stallings. Yeah. So event storming relates to that because what you do with event storming is it's at its heart, it's a very simple technique where you get a bunch of people to map out the life cycle of, um, like it could be a business process and, and what's happening in that business process using sticky notes on a wall. Um, and what's nice about that is you're representing the things that happened in that process with sticky notes.

Paul Rayner (36:13):

So it's something that with nontechnical people in the room, they say maybe people that are experts in extended warranties or something like that. It's very accessible to them as an approach because they can just grab a Sharpie and some sticky notes and start putting ideas up on the wall. And it allows, what I found is it does a really good job of um, highlighting where there are gaps in a, in a process. It allows you to build up a shared narrative of what's going on in a process and help align all the different perspectives that might be in the room on how that process works.

Paul Rayner (36:53):

Both from, like I mentioned, the nontechnical people in the room to the developers that may be saying something like, Oh, well this is, this is where we integrate with this third party system and here are the events that go in and out from that. And um, and so it's a way of doing that in a, in a tactile, collaborative, fun kind of manner. Nobody knows the whole story, but together what we tend to find is that a diverse group of people that are all working in that area can configure a more complete picture of what that narrative is doing it together.

Paul Rayner (37:32):

And so then storming starts with events, but then there's a whole bunch of other sticky colors and things like that, um, that allow you to express commands, read models, uh, interactions with external systems, um, human actors, system actors, um, uh, boundaries or emergent boundaries between things. Um, and, and so like I had a team, um, in just about, just over 12 months ago, I think that we are using Kafka.

Paul Rayner (38:06):

And so they were using, we did an event storming session where we mapped out the flow of events for the business process that they were working on and being able to identify what things need to, what things need to be put on certain topics and what things are internal events to certain systems and where, where should the boundaries be and how do we assign team responsibilities to developing these. So it's a really nice way of building up a visual map of how things are, how things should be and helping the teams design how to design something that's actually going to do the job.

Tim Berglund (38:43):

So it's a way of, well, let me not put it that way. There is this business process you're trying to automate. We'll just call it that. And it's an emergent property or it's an emergent phenomenon from all of the minds that are collaborating in the business. There are these people who are doing jobs and doing things and nobody really knows, like you said, but the process is, it emerges from their cooperation and events. Norming is this thing with sticky notes and Sharpies to uh, pull that out of everyone's minds and make it explicit physically in the room.

Paul Rayner (39:15):

Right? Make it explicit physically in the room as a single narrative where you're laying out the sticky notes, expressing a timeline from the earliest events on the left side of the room across usually a blue sheet of butcher's paper to the latest events on the right. And being able to show that as something of a consistent narrative and where it's not consistent being able to or where there are gaps, being able to expose those. So have those kinds of discoveries about assumptions people were making that perhaps aren't incorrect or where there are disagreements.

Paul Rayner (39:51):

Having those exposed early, you know, so you don't want a big design up front, but big discovery upfront is your friend. So being able to, you know, do optimize for discovery. Um, Dan North a few years ago talked about, you know, imagine you're working with a team and it takes them 12 months to develop an application and then you ask them, well how long would it take you to redevelop that from scratch based on what you know now? And most teams would say, well, a significantly shorter period of time, like maybe three months or something like that.

Paul Rayner (40:26):

And the idea there is that when nothing else has changed other than their knowledge level. And so the real constraint that the team is under is the ignorance of what it is that they need to build and, and how they need to build it. And so the idea of optimizing for discovery is that you're trying to identify those areas of ignorance and get them out early and be able to deal with them and then reflect, you know, make decisions at the right time and, and then take the, the richness of understanding that's coming from that and express that in the software.

Paul Rayner (41:04):

That seemed a little long winded on my part, but hopefully gets the idea across.

Tim Berglund (41:08):

Mine. Mine was shorter, that's for sure. Yeah. You're, you also wrote a book on this and led workshops and how to do this. It's arguable that you know a little bit more about event storming.

Paul Rayner (41:21):

Well, and I feel like I'm sort of trying to explain basketball. Right? You know, people that haven't played it yet. And sometimes you just need to say, look, you know, watch this video, um, or you know, here's a demo of this thing. And, and then they can kind of see it and then try it for themselves and get a better sense of that. So the idea with the book was to, you know, um, people that want to get, facilitate some kind of workshop like this to help them get up and up and running as quickly as possible with being able to do that.

Tim Berglund (41:52):

That makes sense. Now, last, uh, what is this thing? Question, event sourcing. Now I know there, we'll start with a word event and then a word that starts with the letter S and they have nothing to do with one another, but, so everybody, it's not an alternate form of events norming. Um, right. But it is another thing that comes up a lot in the community. So, uh, and I think it's a little bit more related to Kafka. So what, uh, what is event sourcing?

Paul Rayner (42:21):

So, if I, I mean the simplest way I can describe it is when you're thinking about the last cycle of some kind of entity, the things that, and what I mean by that is the things that can change over time. What events are seeing says, let's model each one of those changes as an event. And, and then if we want to find the current state of that entity, then we're going, what we do is we play through the events to figure out the current state of the entity rather than necessarily storing the state itself is the first-class citizen.

Paul Rayner (42:55):

So what we're interested in that case is that the changes over time rather than, you know, the current state. Um, and there's a whole bunch of benefits to taking that kind of approach to um, implementing, you know, the life cycle of a particular entity over time. But that, that in an, you know, that's the elevator description of events or seeing

Tim Berglund (43:23):

Yes. And that, actually, I think that elevator description is good for now. I mean that's a topic for a, for a whole right side cast easily. But it, it, uh, it sounds an awful lot like the database inside out, uh, architectures that have emerged around Kafka. And that's the thing that we've talked about on this podcast. And you can Google that phrase and you'll get to articles and books, uh, quickly. In fact, I'll make sure there's a link in the show notes to some, uh, uh, resources there.

Tim Berglund (43:53):

But, uh, events sourcing is a thing that's been around for a while and they are kind of pre-Kafka. It's of data infrastructure that have been intended to support that. And I, I know the purists in that world when, like when I talk about, you know, here's how to do this with a topic and I got Kafka Streams, KTable, they get a little twitchy, you know, like that's not orthodox event sourcing. But, um, how our event sourcing people like almost always DDD people, not necessarily, really.

Tim Berglund (44:24):

Okay. I see it as this conch correlation, but there is a strong correlation and it is true that a lot people in the DDD

Paul Rayner (44:32):

Community have, uh, implemented some kind of event sourcing system at some point. And, and then there are people that are doing events or seeing that, you know, there's some people that are doing events or seeing that kind of feel like DDD has added a lot of buggy baggage to what they're trying to do. Um, and you know, there's some confusion sometimes around the notion of what an aggregate is and how that fits in with, um, event sourcing. And I think there's still a lot of work to be done to get some of that sorted out.

Paul Rayner (45:06):

The, um, the idea with event sourcing though is it provides a lot of benefits to being able to track the history of what's happened over a period of time, but it, um, but it's a different style of thinking about coding and the way you approach managing state. So, uh, you know, people have to get familiar with that to be able to, you know, it's like anything, right?

Paul Rayner (45:37):

You have to have a certain level of familiarity to be able to move forward with something successfully.

Tim Berglund (45:43):

Oh, don't I know it. That event driven. Getting, getting people to think in terms of events is difficult because as I said before, most of us are in the process of building at most our first such system. And you know, if you ask people how many monolithic applications backed by relational databases have you built in your career and you know, some ask some mid-career person that and you know, they'll kind of stare off the horizon a little bit and think and not really be able to count how many, uh, but everybody's, everybody's sort of doing their first and it's a very different way of thinking.

Tim Berglund (46:18):

Um, but I wonder, I mean, you, you, um, and we don't clearly script this, um, where I wanted to get to was, it seems like those...

Tim Berglund (46:30):

Look, I, I know there's some, uh, suspicious prevalence of domain-driven design people in the Kafka community and I know there's some connection between event sourcing and domain-driven design. Like you're more likely to find people who are into DDD if they're into event sourcing or into event sourcing. If they're into DDD. There's like a, yeah, I would agree with, I always want to say cool, more comorbidity there. Why, what is it? And this is, this is why I'm talking to you and, and uh, well there's all kinds of reasons to talk to you.

Tim Berglund (47:04):

This is an interesting topic, but why we're talking about domain-driven design on streaming audio is it seems to me that one enables the other, and it seems to me that given a commitment to domain-driven design, why is it easier now to make the jump to event-driven architectures? I'm positing that it is a fallacy, the complex question.

Paul Rayner (47:25):

I think it would be easier for me to, you know, in the time we have just speak to event-driven architecture in general rather than event sourcing. And that might be more interesting to the more precise it gets, the right gets the complexities and orthodoxies of event sourcing, uh, out of the podcast. So I think that, so my opinion on that is that what DDD brings to the table is this concern about modeling concepts in a way that makes sense and managing that over time.

Paul Rayner (48:00):

So there's a lot of attention paid, paid to uh, boundaries and semantics and um, how concepts relate to each other. And if you are trying to do an event-driven architecture and think about what those events express and how events from one context should be translated to another, then you're firmly in the realm of modeling. You're firmly in the realm of trying to figure out how these things relate to each other.

Paul Rayner (48:31):

And that's what DDD really brings to the table is that higher I guess that higher-level attention to um, what do we call things, how do we group things? Um, uh, how do we translate from one thing to another and in an ineffective kind of way? How do we map that to team boundaries? All of those are the kinds of things, um, that, uh, conceptual things that DDD provides a lot of useful tools for.

Paul Rayner (49:01):

And those are all things that when it comes down to event-driven architecture, that if you don't get those things right, then I, you know, no amount of infrastructure is going to save you, right? If you have made some poor decisions along there, and I don't mean that in a really, I'm being hyperbolic, but um, you know, getting the boundaries right and if you get them wrong, being able to evolve them over time is a critical aspect. All, all of this regardless of what technology choices you're making under the, under the hood.

Tim Berglund (49:35):

My guest today has been Paul Rayner. Paul, thanks for being a part of Streaming Audio.

Paul Rayner (49:38):

Hey, thanks Tim. Appreciate it.

Tim Berglund (40:40)

And there you have it. I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can always reach out to me at @tlberglund on Twitter. That's at @tlberglund, or you can leave a comment on a YouTube video or reach out in community Slack. There's a Slack sign-up link in the show notes if you want to register there. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold.

Tim Berglund (50:07):

And if you subscribe through iTunes, be sure to leave us a review there that helps other people discover the podcast, which we think is a good thing. So thanks for your support and we'll see you next time.