Using Event Modeling to Architect Event-Driven Information Systems ft. Bobby Calderwood

Tim (03:38):

At one point, it was hard to build a system on top of a database and build a schema and all that. You had to learn that. … And I feel like it's hard then to think about systems in terms of events. What do you think of that?

Bobby (04:53):

No, I think that is really hard and it's both hard to think about and hard to communicate about. And part of that is the issue that you brought up exactly, right? Part of it is engineer training, right? We were trained by brand new hatchling engineers to think about the third normal form and the entity-relationship diagram and the database. And then sort of projecting that up a layer with a whole bunch of impedance mismatch into our code and thinking about the objects and their relationships and all of that stuff. And that look it's an unnatural way of thinking to your exact point. It took us a lot of training to think about things in that way. But to a man of the hammer, everything seems like a nail, right? Once you have that pattern and set of paradigms in your brain, that's the way that you sort of look at the world and it becomes sort of a worldview and it colors every problem that you approach, etc.. So I think at this point, the difficulty for the engineering side of the organization is exactly what you said. When we're kids, we're used to thinking about things in terms of events. We think about in narratives like, ""Hey mom, I went outside and this happened and then this happened and that causes this other thing to happen."" Right? That's the story we tell our mom when we come back in from playing. So, it's easy for us as humans to think about systems or to think about events that way. But then all of this engineering training kind of gets in the way and we were trained in a very unnatural way to think about problems in this very sort of Silicon-first sort of way, right? So I think for the engineers, that's what gets in the way is our training. On the other side of the house the business stakeholders and the potential users of our systems, the things that we build, I think we have trained them to think about these things wrong too. We train them to think about where's the data? How's it stored with? What's the database? How do I curate it? I think it's more natural for the business side of the house to think about things in terms of events and narratives and the customer comes and he asks, or she asks us to do this, and then we think about it and then we do this other thing instead, right? So, I think the business side of it still thinks in terms of events, but then when they start to think about the systems, they're like, ""Oh, okay, now that event-driven thinking breaks down but it's not this explicit in their brains."" But this is sort of the conversation they have with themselves is I can think about it like a normal human up to this point and now I've got to talk to the engineers and now we've got to talk about it in terms of databases and the stuff that I sort of understand in a very fuzzy way if at all, right? So, I think that's the issue. I think it's a self-inflicted wound on our part. I think we've done this to ourselves. Fortunately, I see light at the end of this tunnel. I think as we do start to sort of ensuring the business narrative as the thing that we think about and then the systems we build, the business processes we build, et cetera, are subservient to that, right? They are built in the service of this business narrative. I think as we get back to that as an industry, it'll be a lot easier for new engineers to come on into the industry.They won't have to unlearn a whole bunch of bad thinking. They'll be building event-driven systems from the first day, and they'll be able to think about it in those terms. So that's kind of the thinking part of it, how we envision and talk to ourselves about these systems, the way we talk to other people about these systems is also really hampered by this, right? So there's this constant problem in software engineering. Agile tried to solve it. User stories, jobs to be done, behavior-driven development, they're all efforts to solve this. How do we capture the requirements of the system? How do we describe the system to each other in a way that the regular humans can talk to the engineers and the engineers can talk to each other and talk to the UI UX people and talk to the architects and... Right? So, how do we describe the system to ourselves during the design phase before any code is written? As we're writing code, how do we imagine that code to ourselves and describe it to each other? ""This is the part of the system i'm working on today."" And then when we're in operations, if something goes wrong there's an IT incident, how do we communicate that on the incident call? How do we talk about what went wrong on the incident call without having the proverbial Brent from Phoenix Project on that call, do we have enough context and system understanding on that incident call to be able to address that incident? Right? So throughout the software development life cycle, there's this pervasive communication problem about how we envision these systems and how we describe them to each other. And there's no good solution, right? I mean, there's been tons of attempts throughout the years in the industry to try to solve this problem and it remains unsolved, right? I mean, user stories at some level are great to sort of during the agile kind of planning and design process, but they're not going to help you during an incident call, right? You're not going to bring up a user story in the incident call to try to say, ""Yeah, this part of the system is broken."" So, we lack this holistic system understanding and a holistic way of communicating about systems. And it's a problem.

Tim (10:06):

There is wonderful mechanisms, but as you say, there are things they don't do. I think the agile answer to that is that you shouldn't want, or it's somehow a quest for an illegitimate thing to try to find something like a story that you can bring up in the support call. It doesn't exist. The code is all that exists at that point. Probably shouldn't debate that right now. We may agree anyway, but what else? I mean, UML comes to mind, and that's... I think many people are thinking that right now.

Bobby (11:44):

You bring up a good point with sort of the agile answer this... In theology there's sort of a parallel argument, right? Is it scripture or code only? That as the source of truth, or is there scripture and tradition? Which is code and tacit knowledge in the minds of the development team, right? It's a big question, right? It's a sort of fundamental epistemological question that arises in software engineering but also arises in these other domains of human endeavor. How much can we know? How do we know it? We know it? This is epistemology. It's the field of philosophy where we talk about these things. it's a huge problem. It's one that has befuddled philosophers and theologians for many centuries, right? And I don't think we'll get there with code, especially not on a 45-minute podcast either. So we're not going to solve this question. there have been these really... And I certainly don't want to come off as denigrating any of these things we're talking about. Look, they're engineering tools. They all have features and they all have trade-offs, right? And I'll probably emphasize the trade-offs because I'm looking for a better way, but the UML community, you sort of alluded to it earlier, some really fantastic work. I mean, some almost magical engineering to your point, this ability to round trip from code to diagram. That's magic, right? That's basic stuff. User stories, jobs to be done, behavior-driven development was sort of given when then Gherkin and phrasing and all that stuff, all super great attempts to solve this fundamental problem, this communication and no ability problem about how we collectively understand a complex thing. There are some drawbacks, right? So UML, I think for the problem that I am discussing where you're trying to get this holistic system view across multiple skillsets, right? So you want your business stakeholders even potential users, UI UX folks, software architects, project managers, software engineers. You all want them on the same page, understanding the same things and knowing the same things. UML is not for that, right? UML is very focused on engineers. It about specifically sorts of object-oriented programming disciplines. There are parts of UML that are not specifically object-oriented obviously, but a lot of it is sort of geared towards object-oriented code design right down at that sort of code level. And it's really by engineers and for engineers, it's not about communicating how the system works to everybody else, right? Because you show a sequence diagram or some of these other things, you show that to other people and they're like, ""How does this help me make more money?"" Right?

Tim (14:38):

It is a specialized form that you actually have to learn not just how to create, but how to consume.

Bobby (14:44):

And at that point you might as well be writing code, right? Because it's by engineers for engineers, it's only consumed by the engineers, right? At that point why not just be writing code? At some level, maybe it helps you visualize it because then it sort of engages that visual cortex, that part of your brain is super well developed.But as an industry, I think we've mostly left you all behind. There's a few artifacts, like sequence diagrams that sort of assist in kind of modern agile software practice. But a lot of that stuff has gone by the wayside and to your point kind of for good reason, right? It adds overhead to the engineering process without this sort of holistic communication benefit of being able to describe what we're doing to non-engineers. Then there's... So that's kind of by engineer for engineer type of diagramming and modeling disciplines. On the other side of the spectrum, there's sort of this low-code world where our IT departments are a pain to work with. And so we're going to end-run around them and empower this sort of a legion of citizen coders to draw a picture and then press a button and then that picture turns into a system. And all of these things are sort of in these black box runtimes. And so from the IT department, it's sort of a double slap in the face, right? The IT department is like, ""Okay, so you're end-running around us. Cool. That feels really great. And then you're going to dump onto our laps this black box platform that doesn't fit any of the ways that we do any of the rest of this stuff, right? It doesn't integrate with our cICB pipeline. It doesn't run on the same hardware, the same platforms. We don't know how to keep bugging. When something goes wrong in production, we don't have the same level of monitoring and instrumentation and stuff that we're used to with our regular real coding language things like the JVM or whatever we're running on."" And so now it's sort of this double slap in the face like, ""Okay. A, you showed us that we're hard to work with and you'd rather just write the code yourself. Okay, great. And we have to operate this thing that is sort of pain and that doesn't fit with the rest of our IT world."" So I think both of those fall short of the sort of whole system understanding and communication and bringing together kind of in that DevOps spirit, bringing together the different parts of the organization to work in harmony and build common understanding. But I think the low-code is a bit of a nuclear option, right? Like, ""If you guys don't get on the same page with us, we're just going to forget about you and just go do low-code. Do you know what I mean? It's more of an alternative than it is a thing that brings parts of the organization together in my view and in my experience. So in my topic in my on the talk, I mentioned this a little bit. On the one hand kind of in the old days, we had this big design upfront moment where we're spending a lot of time on the design. And then as we moved over to agile because a lot of these waterfall type processes didn't work, they postponed risk and all the things that we don't need to go into now about why sort of waterfall processes don't work for a lot of types of software engineering. As we swung over to Agile, we sort of threw out that design discipline in a lot of ways. We sort of went from big design up front to no design up front and now we're just sort of like, ""Oh the design will emerge as we test code refactor."" Test code refactors. A design will emerge in our code. I've never seen it happen so I don't believe that story. Maybe within a single code base when you're talking about how you structure your classes and function calls and method calls and stuff within a codebase, maybe that can happen to be a TBD, but not a distributed systems design, not a, like I said, sort of a holistic and well-thought-out way of solving business problems. That's not going to emerge from any particular code-based test code refactor cycle, right? You have to have a design, right? And when software engineers code in the presence of a design, they are a lot more efficient and they're able to solve their problems a lot better. And they spend a lot less time thrashing in the, I don't even know if I'm close to solving a problem because I don't know what problem I'm solving, mode. Early in my Agile software experience back in the late '20 tots and early '20 teens, I spent a lot of time as a junior developer in that mode where I don't know what problem I'm solving and so I can't really measure if I'm making progress. I'm doing stuff and I'm keeping busy and my head's down and I'm reporting stuff on standup, but I don't know if I'm helping or if I'm hurting. So those are the two extremes. The big design upfront and the no design up front. I think both are really flawed. So in the search for this lightweight language that will allow us to do some design up front, but sort of just enough design upfront. And that serves us downstream with design and implementation and operations in the ways I described getting everyone on board, getting the ops people and the business users and the stakeholders. Kind of getting everyone speaking the same language. The best thing I've found in my career to this point is event modeling. Event modeling is a discipline that was invented sort of pioneered by Adam Dymitruk with help from Greg Young. Greg Young is sort of the coiner of the term, event sourcing, and CQRS. He sort of invented that at some level. He's very, very good. In fact, I was just on a meetup with him a couple of weeks ago where he gave his sort of original event sourcing talk. And it's sort of this talk that he's been giving for, I don't know, 15 years, 12 years at least. And it's about a financial domain. So that kind of really spoke to me from my past experience. And he talks about why event sourcing is superior to thinking about the current state. Thinking about the events really helps you in a lot of these ways. So Greg and Adam work together. They're kind of up in the Seattle and Vancouver geography. And so they've been fellow travelers for a long time. And then Adam sort of distilled there is very extensive engineering experience, sort of distilled a lot of these thoughts around CQRS, event sourcing, how to do it in practice, what does it mean for coordinating with other teams and describing the system to your users and how do you incorporate the user journey into this thing that is sort of backend in terms of event sourcing. So, he distilled all this down to this really beautiful and very simple modeling language called event modeling and it's super simple. And that's really the benefit of it is its high level enough and descriptive enough for non-technical skillsets to be able to engage and contribute meaningfully to the design and sort of system understanding. But it's also well-specified enough to be immediately actionable by the software engineer. So, it really is a sort of Goldilocks zone of just enough design, general enough for non-technical skillsets but specific enough for software engineers whereas UML is much too specific and doesn't speak to this nontechnical skillset. So it's this really, really great distillation, I got to give Adam a lot of credit for what he's built. So I discovered this while I was doing professional services work. I was doing an event storming, which is sort of an ancestor technique to event modeling. Alberto Brandolini did event storming, wrote some really good thought there as well, came from the DDD community. And I had come up with my own kind of hackneyed simplification of event storming that was more... Event storming tends to be very high level. How do you map out kind of a whole domain? I only came up

Tim (22:32):

When you're trying to explore a domain and generate shared understanding about-

Bobby (22:36):

That's right. And event storming explicitly is sort of about exploring the problem space. Like, ""Hey, what's the problem we're trying to solve? How do we all work together to solve that problem?"" And it's much less about solution space, whereas event modeling is really truly solution-oriented, we're speaking out a piece of software we're going to build when you're approaching event modeling rather than exploring a business domain. And so I had to come up with my own kind of hackneyed simplification to event storming. I called it a low level or whatever. I don't know. It was a terrible name, but low-level event storming, and we were doing that... When I was at Capital One when I was at... As soon as I started evidence systems and doing my professional services work with a bunch of big companies doing this, that's what we're working on. And in fact, actually, two colleagues of mine and I at Capital One were sorts of given the job of coming up with the microservices curriculum there and we incorporated event storming as sort of an explicit part of the two-day training that we developed. We developed the curriculum and talked about all that. So event storming I've been doing it for years since I don't know, 2016, 2015, 2016 timeframe and it was great and I really liked that. As I found event modeling I was like, ""Yes, this is what I've been looking for. This is the kind of simplification, the lower level kind of zoom in and starts talking about a specific system kind of focus that event storming didn't give me."" So it's great and so I've been doing it ever since kind of 2017-2018 timeframe.

Tim (24:06):

I want to revisit, just because you're making me think things and not because this is the time to debate it, but the question of agile architecture. You have a settled opinion that it is substantially not a thing. I might go and invite some people to just talk about that because it's interesting and it kind of relates to this.

Bobby (24:38):

Just anecdotally and personally, I've met software architects or enterprise architects at companies that adopt agile and they feel hopeless. They feel disempowered and disenfranchised at some level where they're like, ""I used to know what my job was and I drew pictures and I thought about latencies and SLS and that's what I used to do. And now in this agile world, I don't know what I do. I serve as sort of a consulting developer on teams and I sort of visit three or four teams and help them with hard things. But that doesn't feel architect to me. That sort of feels engineer SWAT team to me."" Right? So I've seen a lot of architects struggle with this. So, I think it's an open question. I don't think we've solved it all the way.

Tim (25:25):

We haven't. I'm now just imagining the agile architecture proponent answering what you just said with, ""Good.""

Bobby (25:31):

All right. And that's fine. If that's your answer, that's fine. But look, there's a real human cost there, right? I mean, you've got these people who have been doing a job for a lot of years and they have a ton of wealth of experience and depth and they understand. They know where all the skeletons are buried in these systems and to disenfranchise them and lose them, you're losing a lot. There's a real human cost there.

Tim (25:57):

... there were once a lot of blacksmiths and there are fewer of them now. Sometimes that happens. And there's a human cost and somebody's got to love those people but in economic transitions like this one could argue that it is such a time for the enterprise architect, but maybe not. It's very much an open question and that would be an interesting panel to do. So for the future, if you're listening at me, if you think that sounds a good idea and where else do I want to go with that? We talk about architecture and... So, the other insight I had has to do with a book I'm reading right now called The Fifth Discipline by Peter Senge. It's a systems thinking. It's really a management book, I think, but a super intellectually interesting one. The subtitle is The Art and Practice of the Learning Organization. It was originally written 30 years ago but updated maybe 15 years ago. So some of the examples feel like I'm young again, but the ideas are kind of blowing my mind and the whole architecture emerging thing, and on what scale does architecture emerge and you made the point here and in your talk when you've got multiple systems that are interacting. It doesn't seem like the agile architecture thing really happens then. Maybe if there's one system, there's enough mental state that implicitly or otherwise people have the structure of the thing in their minds, they're able to reason about it, but then there are these...You're building a composite system of individual subsystems that have fairly primitive interfaces and all you really know is that primitive interface but not enough of the structure and interaction of the other system. And if you never know enough about the structure and interaction of the other system, then it's maybe impossible on principle for that architecture to emerge.

Bobby (28:00):

And there's a lot of really great writing and thinking about the dynamics of complex systems, etc. I really love the DevOps community for embracing and really John Allspaw and some of those other folks that look at complex systems and complex system failure specifically. And they're like, ""Look when a complex system fails, you can't just fire the person who was the proximate cause, because it is not their fault."" I like it if you look at the dynamic- And there may not be such a thing as a fault, but complex systems are definitionally complex. There's a lot of different concerns and they are twisted and intertwined with each other in ways that make it impossible to predict.That's the definition of a complex system is you cannot predict the outputs of the system from the inputs, right? That's how we define a complex system, and... So to that point, it is unethical to fire someone because they put in the wrong inputs and got an output you don't like because that's not. The Five Whys Analysis and all of that. That works at some level, but not in an absolute sense when you're dealing with complex systems. The real trick to this is... Look, and this is the trick for political science and political economy too, right? Where you've got this infinite possibility, all of these economic agents interacting with each other. And a lot of the times that turns out really well and capitalism, trade, and markets, I'm a huge proponent of those things. For good reason, I think we've seen historically that those things work really well and they work better than a planned economy. So there's this real tension in these complex systems between teleology, which is like, ""I have a goal and I want to get to that goal."" So that's kind of teleology. We're doing a lot of philosophy on this podcast. So beginning with the end in mind there's a lot of management thinkers that talk about that. So teleology on the one side and I put the kind of UML and a lot of these big design upfront kind of waterfall things in that teleology first camp, teleology is being the most important thing. We want to know where we're going. And then there's sort of the emergent... Hey, good things happen when you release the constraints in the system and allow the agents in that system to interact with each other however they choose. A lot of times those systems will come to a steady-state that is superior to anything you could've arrived that teleology. So like a planned economy is teleological, a market economy is emergent. And again, historically I mean we don't want to turn this to a political show, but historically we've seen that markets are superior in many ways to planned economies because the unforeseen steady states that our market economy can achieve outperform the narrow of a planned goal that I had for this planned economy. There's something. There's a real similar dynamic in software where there's this real tension between where I want to go and if I have too many constraints in the pursuit of that goal, I eliminate the possibility of these really good steady states that I couldn't have envisioned beforehand, but that would be a much better place for my company to be than the goal that I set at the beginning, right? So it's this real again. Epistemology, teleology, it's this real thorny question. You don't want to lock down that process too much. And I think that's why a lot of people from the agile community would say, ""Good, get rid of those enterprise architects. Those guys were really cramping our style, right? They're harsh on our mellow. They put all these constraints on us and as a result, we couldn't do our best work."" And that's fair. I think there's an absolute truth to that at some level. When you put constraints on a system, especially if you put the wrong constraints on a system, you end up tamping down a lot of those hypothetical potential good steady states that that emergent system could obtain and achieve. On the other hand, if you have no rules of the road, if you have this sort of pure laissez-faire let everyone do what they want the thing you could end up at sort of a local optimum, but vastly miss sort of the global optimum. So you've got to figure out a way to sort of set the right rules of the road, set just enough constraint and just enough rules to promote the incentive structures and the emergent states that you want and eliminate some of the emergent states that you don't want or that would be very suboptimal. And that's the role of regulation in the economy. But doing that well is a dark art, right? There's no scientific way necessarily to sort of think about these things and pick among the outcomes of these complex systems that you want and eliminate all the bad ones, right? You can't really do that because again, the nature of complex systems. But you can set reasonable rules of the road that are based on values, that are based on principles, right? Not a tactical like, ""Do this, do that,"" kind of a mechanical set of constraints, but rather a set of constraints that are around values and around principles and around the kind of these broad ideas that will guide us to the right set of emergent states in this complex system. Well, we are really deep there for a second, bringing all the way back to event modeling. I think that's what we have here with event modeling is this sort of... Let's just set a couple of rules of the road. Let's put the business process and the business events and the narrative. Let's make that the central thing, right? Not the needs of the Silicon or the training of the engineers, but the business process. Let's make that the center of concern. That's kind of principle one of event modeling. So, when we do event modeling, we talk about the events first. So in event modeling, there's kind of these four visual words. There's commands, events, read models and interfaces or wireframes and that's it, right? There's just these four words, and there's a set of sort of structured data flows among these kinds of components, so it makes this sort of graph. And the first thing you do in the practice of event modeling is just to get the events up there. So they're represented by these orange sticky notes and you get them up and sequence them into kind of a plausible narrative. And you say, ""Okay, these are the major business outcomes that we need to record in order to understand what's going on here."" So we talk about how to record those things in terms of events, and then the other sort of principle or guiding value in event modeling is let's talk about the user's experience. Let's think about their flow. Let's think about what they experience. So, the next thing we do in our event modeling practice is we talk about the wireframes. So we wireframe out the system. It's similar in principle to journey mapping, user journey mapping and customer journey mapping, but you think about what the users are experiencing. Okay. At this rough part of the chronology here, the timeline is running from left to right along the bottom. At this point in the event chronology, what is the user seeing or experiencing or doing? And we sort of map that out. So those are two of the big values. Let's think about the business narrative and let's think about the user's experience. Those are the two big values that set the rules of the road for event modeling, except these are the basic structures for event modeling. And then in order to get well specified and be able to talk about specific software components we need to build, then we talk about commands and read models. So commands are how users introduce change into the system. It's, ""Do this for me, please,"" and the user clicks a button in the wireframe that sends a command to the system saying, ""Hey, please do this for me. Place my order. Sign up for my account. Transfer money from one account to another."" That's a command. And then that command we think about it and we process it and we reason about invariance and say, ""I don't know. Do we have an account with this customer? Does this customer have an account with us? Do they have enough money in their account to cover this transfer?"" We assert all the invariants there when we're sort of processing this command. And as a result of that command, we either return back to the interface like, ""Nope, sorry. We're not going to do that."" Or we write out an event and say, ""Yep, money transferred. That event has happened. We're going to write it down."" And then...So, that empowers our users. We empower our users by putting the commands on the board so that they can make a change. And then we inform our users via the read model. So as those events are happening, we aggregate those events up using technologies like Kafka and Kafka Streams especially or KSQL.One might need such a thing you build this reading model in order to answer your users' questions about the state of the system. I place my order and then I go check on the status of my order. The read models, what it informs me about. Has my order shipped? Is this order the right one? And all those questions that users may have. we empower, we inform our users. Those are the other kinds of values that shape this. we're focused on the business narrative, we're focused on users' experience. We empower our users, we inform our users and then we also need to add to integrate with the outside world because our system doesn't exist in a vacuum. we need to be able to export or import state from other systems. There's a pattern in event modeling for external state import where we listen to the events of some foreign system and then incorporate those events into our own narratives and then take some action on that. That's all about listening to some foreign stream. Again, Kafka streams and other related stream processing technologies are great for this because that upstream system doesn't even need to know that I care. They don't even need to know about me. They're just doing their thing going about their work the way that they're supposed to and generating events as a result. And then I can subscribe to those events and import their state into my world and inform what's going on in my system.

Tim (38:17):

That can be some terrible thing. You're scripting a file or something where you're doing CDC on a table, whatever it is. The mechanics of the integration happen and you turn that into events and there's a sense in which you have to maybe make sense of that external system's event in terms of the context of your system, right?

Bobby (38:36):

That's right. So we call that translation. Adam calls that translation in his world. So if you're getting low-level events, so his canonical example is geo events. So your phone is just slinging off these geo coordinate events because it's kind of creepy and striking you everywhere you go, but it's throwing off these geocodes. But those geocode by themselves don't mean anything to your application necessarily. So you have to translate those raw geo events via some sort of business logic to contextualize them and make them relevant to your application. So oftentimes that's something like geofencing where you say, ""Hey when the customer enters my store or when the customer interests the hotel, that's a business event."" So I'm translating these raw events via this geofencing piece of logic and then I emit this more contextual event to my kind of local event topic saying, ""Okay. Customer showed up at the hotel, we should do something as a result of that. Or maybe we just record it and figure out what to do about it later,"" but that's an important thing in our domain. exactly. There's sort of a translation process. And then the last pattern that goes with this value of we don't exist in a vacuum and we need to integrate with other systems is internal state export. I always mess that up. Internal state export where we've got stuff going on in our world. We can't just rely on someone else to listen to us and do the right thing. We need to go out and issue a command into some foreign system, right? So a lot of times it's like payment processors. That's kind of Adam's canonical example of this. Things are happening, users are checking out and doing all their stuff in our application, and then we've got to go talk to that dreaded payment gateway and it's a foreign system and we don't control it and we don't even know if it's event-oriented or not. We just need to go cause some change in that system. So right now, I use Stripe in my application event building right now, which we'll talk about in a minute. And that's how we treat Stripe exactly. We have stuff going on in our application. We reach out and talk to Stripe. Stripe is fantastic. And they do have an event-oriented interface, which is their webhooks. And then they basically call us back and say, ""Yep. A good happened,"" or, ""Oh, oops. There was an error, a bad thing happened."" But we get an event back as a result of that. If you don't have an event-oriented foreign system, you just sort of call that command or web service or whatever. And then the result of that web services call will include a lot of information about whether it succeeded or failed, and then you write that down as an event locally as well. So we're still always thinking in terms of events, even when we're dealing with foreign systems sometimes we have to generate those events ourselves as a result of the web services call, whatever the result of taking that action was, but for good foreign systems like Stripe we can call them and count on the fact that they're going to sling an event back to us

Tim (41:19):

Good external foreign systems that do some sort of financial transaction in an event-driven way. I don't know if you know anything about that, but I know a guy I could introduce you to who could tell you all about it. And just if you're listening and didn't pick up on that, that guy's Bobby. He's the guy who knows everything about that. So anyway I imagine in the external state import and internal state export pieces, I mean, those get reduced to code at some point when you're interfacing with things that are not so well-behaved, meaning things which are synchronous, that's where you get those little impedance matching transformers doing polling or keeping tables of things and doing the adapting from the internals of the event-driven system to the unfortunate synchronous externals of the world.

Bobby (42:16):

And look, that's a reality and any solution which has to be a big bang, and you have to apply it all at once across every piece of your organization and rewrite every piece of code you've ever had, it's never going to work, right? That's not a viable strategy. So event modeling acknowledges sort of the realities of that. And in the internal state export pattern, Adam explicitly talks about, you've got these events going by, you actually build up a read model solely for the purpose. The read model is not for informing your users, this reading model is just kind of the to-do list of work that you need to do against some foreign system. And you may be able to do that piecewise. Each event that flows through you do your side effect and then you record the result of that side effect. But look, sometimes you've got to batch those things up and send them once a day as a file or...It's just some ugly thing, right? But building that read model is sort of a key part of that internal state export pattern because having that read model allows you to reason about the SLA and reason about are we doing this in terms of a single piece flow of each event or are we matching them up and sending them in micro-batches? Or are we doing a big file at the end of the day and we're reconciling some external ledger or something? These patterns are sort of sufficiently flexible in general that allows you to do a lot of things at the implementation level, which satisfy the design, but are ugly and dirty and you've got to sort of hack around the fact that you're dealing with some system that doesn't deal with this very well. Adam has a whole article about event modeling for legacy systems or event modeling for existing systems, where he talks about you can take an existing system and then start to sort of model it out, what it does today. And then start to sort of change and evolve that model and slice off features from that core system and start doing it yourself and building these little event-driven slices of the event model outside of that foreign system. While still relying on that legacy system to kind of do the bulk of the work or some part of the work that is regulated or whatever you need it to do. Even in that future date that we talked to at the beginning of the show when engineers are coming out of school and the first system they work on is an event driven system and that's the only way they know. At that future date, you don't have to worry about those things quite as much.

Tim (44:52):

But there's the actual process you gave us the visual language, and then there is...You said this almost was born as a souped-up version of events storming and event storming if you don't know, is a thing a group of people does together. There's a workshop process of doing this and then there's the visual artifact. I'd love to hear about where you've done this. What successes can you talk about that you've put this in practice?

Bobby (45:55):

The one I refer to over and over again in my talk is with a big tax preparation service we're doing the kind of helping them re-envision and re-engineer and re-design their tax platform kind of the core piece of their business. And we did event modeling with them. Really successful. We got all the kind of business folks in the room with the engineers and just from a communication and context sharing-perspective, the workshop itself was a huge success and it was great. The artifacts that come out of it we've got a big piece of butcher paper covered in sticky notes. Leave something to be desired, which is why we ended up building our own sort of event modeling platform, which we'll talk about in a second. But the artifact leaves something to be desired. We did end up digitizing that artifact in and making stories out of it and building to those stories. But that process at the end was really hard, but the actual workshop moment itself was great. We also did one with a pharmaceutical research company. They try to find new drugs and treatments for genetic and other types of disorders via this big data processing pipeline. The super cool company just raised a bunch of money, but they're really doing well. We went and did a workshop with them to do exactly this, kind of map out their domain. And part of the day was sort of this high level more event storming kind of exercise. And then part of it, we sort of got down to brass tacks on a particular piece of the system and then talked through, ""Okay, how would we build this piece? And what would it look like and what would the user see?"" And we did that via event modeling. So both really successful workshops. Really loved that. In my own software engineering for building our products internally at Evident Systems, that was all professional services staff. But on the product side of Evident Systems, we also use event modeling. So for our real-time payments system, for our event modeling product itself. We do event modeling, which is super meta and cool, but we actually build event modeling to talk about how to build event modeling in our event modeling platform. So we do all of that kind of stuff. We eat our own dog food with the product itself to do those things. As I said, sort of getting to the end of event modeling workshop and having seen a bunch of people try to take the panoramic mobile shot, a mobile photo of the big piece of butcher paper covered in sticky notes. And then it's two or three BAs for three days just trying to digitize this thing and squinting and looking at other people's handwriting through this blurry panoramic shot. It's not great, right? It leaves something to be desired. Event storming was sort of meant to explicitly be a co-located workshop experience. When Adam designed event modeling, it was a little looser. He was like, ""No, this can be done effectively, remotely."" This can be done effectively with everyone sitting on their own computer kind of in some sort of collaborative environment building this event model. So our experience with some of these successes that we talked about and understanding that part of the philosophy from Adam that this doesn't have to be an in-person workshop, that this can be a thing that evolves over time asynchronously as well we built our own event modeling platform called oNote. oNote.com. It's free. You can sign up for it, please go do. It's great. We love it. It was something that we built just to scratch our own itch and to help the event modeling community. But as we got into it and started building it we were like, ""This has huge potential."" Right? A lot of folks were using a kind of collaborative whiteboards for doing these event models and those are great. They're great sort of general-purpose tools, but oNote is a more constrained modeling environment so you can't end up with an invalid event model, right? You can only draw arrows from things that make sense to other things that make sense, right? You can't make invalid connections- It enforces the semantics, the actual structure in terms of the visuals, the vertical space on canvas is constrained so that only events can go down in the bottom section and only wireframes can go in that top section, and only commands and remodels can go in this kind of center, spinal column section. Anyway, so we constrain the model in canvas, but then behind the scenes of that, we've got sort of the visual projection of the event model, the visual piece of it, but behind the scenes, we're building this data structure, right? It's just a data structure that we have in memory. And then we store it to a disk locally in your browser and using this data structure, we can do all kinds of really cool stuff, right. So we can... So as you move from the design phase where we can do sort of that collaborative design, where we're all working on oNote together, collaboration is a feature we're going to release very soon within the next couple of weeks. Probably before this podcast episode drops we will have a collaborative environment. But then as you transition from the design kind of workshop moment into this implementation phase we can generate code. So we can actually generate based on the commands and remodels and the schema that you associate with those things. We can generate GraphQL, gRPC, OpenAPI Swagger, maybe even AsyncAPI. We're looking at that as well. So we can generate kind of the interface definition code, and then you can take that specification and use it in your toolchain of choice if it's Java or JavaScript or Golang whatever. There are already existing toolchains for ingesting that interface definition language artifact and getting to runtime code. We can integrate with Confluent Cloud. That's one of our targets is being able to say, ""Hey, we've got all these lanes of events here, we can just create a topic for each lane, and then we can put the schemas that we define in the event model into the schema registry and confluence cloud."" So there's a lot of integration level stuff we can do to speed the implementation process along. Another cool one there is rather than having to sort of separate user story generation exercise or sort of process that you do in agile, or ritual, sometimes they call it, we can actually just carve up this diagram into the set of tasks that are required to implement this diagram. Adams calls those slices. They're kind of these vertical slices of functionality that have a well-defined contract. They have well-defined inputs, well-defined outputs, and that represents sort of a discreet unit of developer or engineering work. And so rather than having a separate user stories exercise during your agile planning, we can carve this diagram up, dump it into Jira for you as you burn those tasks down in Jira or whatever you have issues or wherever you track your work. As you're burning those things down, we can show in an overlay or something on top of your diagram, ""Here's the progress we've made on this,"" right? So, we can start to really integrate with the design process, the implementation process, and then during operations, as I said, we can integrate with your data platform, pull out some metrics and monitoring, and actually maybe start to visualize the data flow through your system in real-time. You can start to see people are stuck at this part of our checkout flow, and we can see that on the metrics visually and all that stuff. So there's a lot we can do with this event model that's not just about design. It's valuable from the design through the implementation, into operations. Again, another one is management, right? Something bad has happened, our system's broken. The people who built the system aren't necessarily on the call so we're missing a lot of that mental context. Can we use this artifact, this design artifacts, to reason about where the system's broken to visualize the metrics and see what's happening, and then fix the system without needing the original team that has all that shared tribal knowledge in their heads. Anyway, there're a lot of really exciting things on our roadmap. Right now oNote is sort of just the modeling canvas piece of it but we have a roadmap that gets it a lot farther along and becomes a sort of an integral part of accelerating the software engineering process. It helps you to adhere to these broad rules of the road that we sort of outlined earlier, that if all the systems in your enterprise are just sort of following these basic rules of generating events and informing their users, empowering their users, caring about their journey and having structured ways of integrating with other systems. If everyone's following those rules, you don't need real strict architecture from the outside. Those rules are enough to where you end up with this really great emergent type of thing where people are able to use these data and build extensions and really cool alternative views of these data without having to be so teleological about it, right? You can be emergent, you can be iterative and everyone kind of doing the right thing and you get to this really cool steady-state emergent world, the ecosystem, but you do have these broad rules of the road so you don't just end up with this senseless chaos, right? You've solved a lot of the hard distributed system problems. You've solved a lot of the consistency and all of those difficult things just by adhering to the rules baked into event modeling.