Blockchain Data Integration with Apache Kafka

Kris Jenkins: (00:00)

Hello, you are listening to the Streaming Audio Podcast. And today I've got to start with a single word, a word that's going to excite some listeners, it's going to scare off others, but hopefully for the majority of us, it's a word that spikes some curiosity. The word is Blockchain. Controversial. Joining me on today's show is Fotios Filacouris who's a Kafka expert by day and a Blockchain expert by night, and we're going to discuss what they are, how Kafka and Blockchains are at times similar, at times different, and in some cases quite complimentary. I will say before we get started, there are an awful lot of hand wavy hot air discussions in the Blockchain world, and this is not that. We're going to try and pin down what these data stores do and how they work and what their strengths and weaknesses are.

Kris Jenkins: (00:53)

Before we get started on that, let me tell you that Streaming Audio is brought to you by Confluent Developer, which is our site that covers everything you need to know to build successful systems with event streaming and Apache Kafka. Whether you want to get started or get a deeper understanding of the internals, there's information there to help you. And when you need to get Kafka up and running, take a look at our cloud Kafka service Confluent Cloud. You can get a free account and Kafka cluster up and running in minutes, and it will scale all the way up to crazy enterprise sizes. And once you've got an account, remember to add the promo code PODCAST100 to the billing page, and you'll get $100 of extra free credit. And with that said, let's press the hot button topic button, it's time to talk about Blockchain.

Kris Jenkins: (01:46)

My guest today is Fotios Filacouris, Foti, welcome to the show.

Fotios Filacouris: (01:50)

Kris, thanks for having me, it's past and present for me here.

Kris Jenkins: (01:55)

Yeah, you're a returning guest.

Fotios Filacouris: (01:56)

Returning guest. I don't know what I've done to be so lucky.

Kris Jenkins: (02:00)

I do, you've moved slightly because last time you were on here, you were talking about how you've been doing a lot of work with Kafka and financial services, but you've shifted your career recently a bit, right?

Fotios Filacouris: (02:14)

Yeah, I think that's fair to say. I have picked up kind of a side job as well of podcasting about cryptocurrencies and Blockchains.

Kris Jenkins: (02:26)

That's a hot topic.

Fotios Filacouris: (02:28)

It's a hot topic. If you look at the market, it may be a depressing topic. It depends on how you want to view that, right?

Kris Jenkins: (02:35)

Yeah, at the time of recording, it's a bit of a blood bath, but we'll see how that goes by the time this podcast is released.

Fotios Filacouris: (02:43)

"Correction time."

Kris Jenkins: (02:44)

Correction, yes, depending on your perspective.

Fotios Filacouris: (02:46)

But yeah, so I have picked that up, but to be clear, I do still work a lot in kind of traditional finance as well, day job.

Kris Jenkins: (03:00)

Yeah. So you are a solutions engineer at Confluent doing a lot of Kafka and finance and some Kafka and Blockchain type stuff?

Fotios Filacouris: (03:09)

Yeah. So I'm I guess what you would call a SME or an SME of sorts. So people like to pull me into kind of anything financial related when it comes to architectures. I don't know if that's just the proximity of where I am to New York City or I don't know what it is, but I do work mostly in financial related use cases, architectures, things like that.

Kris Jenkins: (03:35)

Okay. So maybe we should start, because there's a lot to cover in this and some of it's controversial, maybe we should start nice and safe on architecture. From a certain point of view, you could say Kafka is a database. From a certain point of view, you could say a Blockchain is a database. How do you think they're similar? How do they differ? What's the alignment?

Fotios Filacouris: (04:00)

Yeah, I think you could say, and I think Martin Kleppmann has said Kafka is a database that's inside out, right? And similarly, I think Blockchain has similar components to it where when you look at the underlying data structures, they are meant to be immutable right. So when you write a message or an event to Kafka, or you write a block to a Blockchain, it's supposed to be immutable. There's metadata associated to that block or to that event that have been written to Kafka so they have similarities there. I'd say some of the differences is just generally across different Blockchains, how quickly blocks are written comparatively to Kafka. So if you look at Bitcoin, for example, they have to go through a validation and consensus process before writing a block and a new block only gets written every 10 minutes. Obviously in the Kafka world, if it took 10 minutes to write an individual transaction or a message to our distributed commit log, we wouldn't be doing so great in the technology space, right?

Kris Jenkins: (05:13)

We'd either be completely different or utterly dead.

Fotios Filacouris: (05:15)

Absolutely, yeah.

Kris Jenkins: (05:17)

Yeah. But I mean, there are two things going on there, which you can already pick apart, which the first is that you've got this idea of data not being, you submit it's accepted, but it won't actually reach the chain for a fair long delay, right? Because of this distributed consensus process. Whereas, in the Kafka world, you elect a leader and that just writes stuff until a new election.

Fotios Filacouris: (05:45)

Yeah, exactly. So Blockchains have found this to be a pain point, let's just say. So there's two types of models. If people are unfamiliar with kind of how Blockchains work, there's a proof of work model, which is like what Bitcoin uses today and there's all of these open validators that can enter a consensus framework within the network. And then you essentially need a certain amount of them to do heavy computational logic to confirm that what the hash that's been written to the current block and the preexisting blocks within the 10 minute timeframe are all valid, right?

Kris Jenkins: (06:30)

So is it fair to describe that as who can write a new block to the database, whoever sells this technical puzzle the fastest?

Fotios Filacouris: (06:39)

Absolutely. And that's what people would call mining, right? That's kind of like what miners do I guess in this space. So they've been trying to get around that with another consensus algorithm called proof of stake, where you have to actually own the underlying currency to be able to enter as a validator and then a certain algorithmic computation determines whether or not you are chosen as a validator for that transaction. So it in theory will speed up how quickly blocks can be written to the Blockchain.

Kris Jenkins: (07:18)

And consume less work, because if you're racing to be the person that solved that technical puzzle to write a new block, then if there are 10 people in that race, one person wins, the other nine have wasted their time.

Fotios Filacouris: (07:32)

Absolutely. And the algorithm itself, and the reason why proof of stake isn't incredibly popular or even widely used yet, is there has to be other determinations outside of just this person owns the most of the underlying currency so they will be a validator, right? There's a lot of complexities that go into the determination of who enters that consensus under proof of stake. But hopefully once it gets there, it will decrease the costs associated with cryptocurrency and Blockchain related transactions.

Kris Jenkins: (08:15)

So I think the contrast it was saying, correct me if I'm wrong, the way proof of stake works is we say, "Who can write a new block?" Well, a 30% shareholder gets to write new blocks 30% of the time. And the nice thing about that is most of the time we know exactly who can write the next block and they just do it and there's no arms race.

Fotios Filacouris: (08:40)

Correct.

Kris Jenkins: (08:41)

Yeah, so I can see how be a lot more efficient.

Fotios Filacouris: (08:44)

Yeah, it also doesn't require incredibly expensive ASIC mining rigs to do the mining process, right?

Kris Jenkins: (08:54)

Like you see footage of farms of high end graphics cards just chewing through data, trying to solve those puzzles.

Fotios Filacouris: (09:01)

Yeah. You wonder why none of my customers can get any physical hosts in their data centers, it's because everyone's mining Blockchain cryptocurrencies nowadays in the height of this pandemic and stuff.

Kris Jenkins: (09:13)

Yeah. I feel we could slip into the environmental argument, but let's try and stay on technology.

Fotios Filacouris: (09:20)

Sure, sure.

Kris Jenkins: (09:22)

So the question that always immediately raises for me is there is an overhead to doing distributed consensus, what is the benefit that it's worth paying that price for?

Fotios Filacouris: (09:39)

I think, and we had a discussion on our podcast with Kai who you may be familiar with and people in the Kafka community are probably familiar with Kai, and we kind of went over this exact topic, which is why would anyone choose to use a Blockchain for things in general if they could use something like Kafka? And I think the elementary answer, my dear Watson, is that Blockchains are designed for low trust environments. So they are designed to operate in a space where anyone can join, but anonymity is priced as a key facet, right? So anyone can become a validator, anyone can join the consensus network, anyone can write, but since anyone can write and anyone can join, there are deterministic values in the underlying architecture of Blockchains that obfuscate the operators, right? So you'll get an address of someone so every transaction can be traceable, but you don't know who's part of that address necessarily, right?

Kris Jenkins: (11:02)

Yeah, that's that weird paradox of Blockchain that you can always trace transaction because it's completely public and open, but it's also completely opaque because in theory you don't know who the hash is, who owns which hash, right?

Fotios Filacouris: (11:15)

Yeah, and a lot of people within the crypto space have a lot of problems with that, being able to trace their whole transaction history as long as you have an address, right? So they're doing things and patching Bitcoin networks and they're tempting to try and obfuscate more of that transaction history, but yeah, you're spot on, there is a strange paradigm. And I mean, Blockchains exploded because, in my theory, this guy, Satoshi Nakamoto who created this white paper for Bitcoin back in 2009 created a very valuable asset and a lot of other people thought that they could create very valuable things based on the underlying technology of that asset. Maybe a bit cynical, but that is my view.

Kris Jenkins: (12:10)

Yeah. Well, I can sort of see vaguely how if you sort of hand wave, if he created a valuable piece of infrastructure, it's legitimate for people to want to build on that infrastructure. That's exactly what infrastructure's for, right? But I'm often confused about some of the use cases we get on top of that distributed leisure technology. And maybe we should talk about a few of those. So what are the big use cases for a Blockchain, right? NFTs? I guess we have to talk about NFTs.

Fotios Filacouris: (12:42)

Sure, yeah.

Kris Jenkins: (12:44)

Coins like distributed leisures, anything else? Smart contracts?

Fotios Filacouris: (12:51)

Yeah, smart contracts. I think they're kind of rolled into the first two that you mentioned.

Kris Jenkins: (12:57)

Give me an overview, survey the space for me.

Fotios Filacouris: (13:00)

Yeah, so I think the biggest issue right now with cryptocurrency Blockchain technology is the overall utility is lacking in day to day kind of life, right? So you mentioned NFTs, NFTs has kind of penetrated popular culture so people know what NFTs are, they're familiar with what Bored Apes are and all of this Yuga Labs related stuff, right?

Kris Jenkins: (13:30)

Yeah, whether we understand them or not, we're familiar with them these days, right?

Fotios Filacouris: (13:34)

It's not for us to decide, it's up to the gods, right? Yeah, but that is one of the major spaces. Now what I will say about NFTs and if you're unfamiliar with NFTs, a quick primer, they are called non-fungible tokens and they are essentially metadata that exists on a Blockchain that is associated to some kind of an asset. So imagine, and this is the most trivial example, you have a picture of the Mona Lisa on the Blockchain and you can validate with 100% certainty that you have the original copy because you have the underlying metadata or the hash associated to that asset.

Fotios Filacouris: (14:16)

So NFTs have kind of changed the world to a degree because there's been gaming projects that have kind of spurred off of the creation of NFTs which has substantially changed the earning potential of people in developing countries across the world.

Kris Jenkins: (14:36)

Really?

Fotios Filacouris: (14:38)

Yeah, so there are people in the Philippines and in Southeast Asia that are able to make more playing NFT play to earn games per day than they could the local minimum wage within their country. So we may scoff at it in the more developed world at what you can make per day playing these types of games, but for people in the underdeveloped world, I should say, or developing nations, it's monumentally changed how they can earn and make a living, which is quite substantial.

Fotios Filacouris: (15:18)

When you look outside of that space, there's payments and there's some stuff with IOT, IOT related use cases, right? Where you do see Blockchains are attempted to be implemented, but it's slim pickings after that, I got to say.

Kris Jenkins: (15:43)

Yeah, okay. I want to pick up on the gaming thing because I'm a modern guy, I understand the modern world, I can see how on some levels gaming has actually become business, right? There was a classic economics article on the selling of in-game hats on, what was it? It wasn't Battleground, there was some online game.

Fotios Filacouris: (16:12)

Lizard or some kind of Warcraft game or something like that?

Kris Jenkins: (16:15)

Yeah, it was some capture the flaggy kind of battle game, but come to me in half an hour. But you could buy hats in that game and there was a flourishing economy in buying and selling hats and you could actually trade them for real money, right? So yeah, modern world games, playing games, certain kinds of games can be a money spinner and more so than a minimum wage in certain countries, right? I can understand that. But where does a Blockchain come into that? Because that game that didn't have a Blockchain, it just had you can trade it within our database and you can turn it back and forth into real cash. That was enough for an economy. Where does Blockchain come in?

Fotios Filacouris: (16:58)

So where Blockchain gaming has completely changed the paradigm is instead of it's play to game, it's play to earn. So these games are specifically designed to issue you rewards based in the local currency associated to the economy that they create within the game. So whereas what you described is you pay to play some game and you pay to upgrade your character and do several things associated to leveling up in a game and then you get to a level where you start selling items on the black market or something like that, these systems and these Blockchain related NFT games are designed to reward directly the gamer in the underlying currency, which then in return raises the value of their token economy and probably a lot of the tokens that they hold internally.

Kris Jenkins: (18:05)

Okay, so the game makers create a currency and then by incentivizing people to play the game, they make the currency more popular, which they're then planning to sell?

Fotios Filacouris: (18:17)

Or collateralize, or swap for other things. But if you think about it, it's almost infinitely cheap for them, similar to creating software, right? It's incredibly resource intensive on the development aspect from an investment perspective. But duplicating software is almost infinitely cheap to do, right? Similarly in these types of environments, if I wanted to create a new item that's a new NFT on the Blockchain, it can be infinitely cheap for them to mint these additional new components within their Blockchain ecosystem to provide for gamers that stimulates their economy. So it's a fascinating concept, I think.

Kris Jenkins: (19:12)

Yeah, I feel like I haven't quite got a handle on it. It's like almost the trade of attention as an economic tool, which sounds weird. But then you see people making a living being TikTok stars and if they have a million people following them, well, they are basically a television channel, right? If you've got a million followers, you are a small television channel.

Fotios Filacouris: (19:44)

Take it a step further, Twitch streamers, right? People watch other people game.

Kris Jenkins: (19:49)

Attention is an asset you can buy and sell.

Fotios Filacouris: (19:52)

It is, absolutely.

Kris Jenkins: (19:55)

Weird, but undeniable, I guess. Okay, let's move on to IOT. What's happening in IOT?

Fotios Filacouris: (20:01)

Well, this is a little bit more kind of underdeveloped, but I think the idea within IOT is if you wanted to track things, I mean the same reason why people would use Kafka for IOT, they would try and use Blockchain for IOT, which is to aggregate metadata associated with supply chain related things, right? Supply chain related use cases. What do you like in those types of use cases? Same things that you like and the same reasons why you would use Kafka, immutable, fully traceable transactionality of data that's been written, capturing of events. I mean, you see heavy investment from people like GE into Blockchain, for over a decade they've been investing in this underlying technology.

Kris Jenkins: (21:03)

Really?

Fotios Filacouris: (21:03)

Absolutely.

Kris Jenkins: (21:04)

They were early adopters then.

Fotios Filacouris: (21:06)

Yeah, I mean the issue is when we start talking about the legitimization, I don't know if that's a real word, we're going to pretend.

Kris Jenkins: (21:16)

I'm going to say it is.

Fotios Filacouris: (21:18)

Okay, of Blockchain in more traditional use cases, things for business, IOT things like that, you find that companies do not want to use public Blockchains for myriad reasons, right? So they want to develop their own.

Kris Jenkins: (21:39)

So they have their own private Blockchain>

Fotios Filacouris: (21:41)

Absolutely, absolutely, yeah. And I think the challenge when you're creating your own thing is after a period of time, you have to justify technologically whether or not it's the actual best fit for the use case, which I don't think people have been doing a good job of, to be fair, right?

Kris Jenkins: (22:06)

Because if we go back to what you said about the Blockchain being good for low trust environment, that feels at odds with a private Blockchain, we don't trust anyone?

Fotios Filacouris: (22:17)

Exactly. Now, again, the thing about a Blockchain is that not one is created equally to the others, right? So we can't just make assumptions technologically that the same Blockchain that's being leveraged for Bitcoin, where a new block is written every 10 minutes, is going to be the same as another, right? So as Ethereum, for example, is completely different, as some of these side chains that are created, they're more centralized in nature. So the technology and the topology of how you create a Blockchain network is similar to how you could within Kafka as well, right? You can decide how many nodes you want, you can decide, I guess, the consensus algorithm. In Kafka's world, it would be how many in-sync replicas you would want when you're writing to a topic, right? Depending on your use case, you can kind of tweak the transactional semantics of how data is being written to it.

Fotios Filacouris: (23:27)

So I don't know, we don't know exactly in a lot of these private Blockchains how they are being developed and in what aspect, but we can assume that their technology is being used, even if it's not in a low trust type of environment, it's being leveraged for some of those traditional components like the immutability, like the distributed nature of it, right?

Kris Jenkins: (24:00)

That reminds me of one thing. We kind of left out in the analysis of Blockchain Kafka, which I think is very important, which is every block, every event on a Blockchain is cryptographically signed in a way that links it to the content to the previous block.

Fotios Filacouris: (24:19)

Correct.

Kris Jenkins: (24:20)

So if you found a way to change the oldest block on the Blockchain, it would completely change the hashes of every subsequent block. That's not true of Kafka, there is no inherent tie between one event and the next event.

Fotios Filacouris: (24:39)

Yes. So technologically in a lot of Blockchains, you cannot change the genesis block. So that's, I guess, the one defense I will come to the Blockchain.

Kris Jenkins: (24:51)

And if you could, you'd have a different Blockchain, right?

Fotios Filacouris: (24:57)

Absolutely.

Kris Jenkins: (24:57)

You'd be able to tell it had been changed.

Fotios Filacouris: (24:59)

100%., 100%. Yeah. So I mean, I think the reasons why people use Kafka and the reasons why people use Blockchain are completely different, but they do have a lot of similarities. And I think we're seeing this whole startup culture around Blockchain related technologies, right? There are businesses, startups, various companies and organizations that are doing different things, right? Sometimes you'll have an exchange that's out there that needs to modernize their internal infrastructure, right? There's no reason why you would use a Blockchain to do something like web services or to shuttle data around.

Kris Jenkins: (25:50)

I would hope not.

Fotios Filacouris: (25:54)

I guess crazier things have been done.

Kris Jenkins: (25:56)

That's true, yeah. I mean, it's still a very exploratory space, right? People are trying to figure out what Blockchain is best at.

Fotios Filacouris: (26:04)

Yeah, it's an exploratory space. I think if you want to look at things in a macroeconomic way, because I tend to do that at times, the overall market cap of Bitcoin at the peak of its kind of height over this past year was about $1.2 trillion. So if you look at that comparatively, that's like the market cap of, I don't know, Apple, right? So in the greater economy, it's one very large company, right? Bitcoin is synonymous maybe now with what Apple is, but you have all of these other technologies and other Blockchains that are hanging out under it that's very adjacent to what you saw in the .com era where there's thousands of companies that are out there and who's going to actually make their way to the top.

Kris Jenkins: (27:03)

Yeah, yeah. I remember I've seen that few times in this industry and anyone who's been around a while in this industry has seen a lot of money, a lot of speculation, a large proportion of it will just crash and burn, but you never know which players will emerge to be significant, right? I mean, who remembers MySpace and Friends Reunited? Some of us, but who knew that Facebook was going to be the one that would survive, right? And there are probably a thousand other companies that did exactly the same thing that you never heard of. You don't know which Blockchain solutions will survive out of this. And you know some of it's speculation, but which bits.

Fotios Filacouris: (27:43)

Yeah. I mean the main difference I guess comparatively to the .com bubble is a lot of those companies were kind of pre IPO, right? So it was harder to invest in them. Whereas in the crypto Blockchain space, it's a lot easier to speculate on some of these assets because the way you speculate on them is you buy some of their tokens and hope that they go and take you places with them.

Kris Jenkins: (28:15)

Incidentally, if anyone listening wants to buy some Jenkins tokens, I'll happily sell them.

Fotios Filacouris: (28:20)

And I have taken a nice picture of you, Kris, we're going to create a nice NFT, we're going to throw it on the Blockchain.

Kris Jenkins: (28:29)

So while we're exploring the solution space, we can speculate forever about which companies will survive, but there is as well as the Kafka and Blockchain having similarities, differences, you've been actually working on some use cases where they compliment each other, right?

Fotios Filacouris: (28:49)

Yeah. So I think the first use case... So I guess I'll do a primer on this, where Kafka is being used is not like in any type of technical sense where it's like calling a smart contract and passing it to a Blockchain or something like that. Kafka is doing what Kafka does really, really well, which is a lot of the use cases that you'd see even outside of traditional Blockchain related use cases.

Fotios Filacouris: (29:24)

So for example, you have a lot of institutional banking right now that is trying to provide crypto-derivative assets to some of their private banking customers. So I'm a high net worth individual and I want to buy Bitcoin and make that part of my portfolio and I want you as my financial institution to custody my coins and I just want to see the dollars and cents in my balance sheet, right? So Kafka comes into play there when you're doing any type of, it may not actually be calling out to centralize exchanges like Coinbase to make the calls for the transactions or things like that, but what it is doing is doing traditional things like capturing the statuses and scraping those types of events that are going in and out so that you can get a real time update on the kind of the state of your transaction and how it occurred, right? It's capturing the logging associated to the applications that are being built to facilitate that type of transaction and passing them to downstream big data systems, data lakes, things like that.

Fotios Filacouris: (30:49)

So in a lot of centralized exchanges use cases that we're working on, or even traditional banking related use cases where they're trying to offer Blockchain related cryptocurrency to their private banking customers, Kafka is being heavily utilized as a core piece of infrastructure in a lot of those use cases.

Kris Jenkins: (31:15)

So again, there's that capture real time process and notify backbone.

Fotios Filacouris: (31:21)

There's nothing groundbreaking about it. But again, it is fascinating to see the integration, I just think it's so interesting that a lot of the reasons why people like to use Blockchain technologies, Kafka kind of does it better in a sense because it's faster, I don't want to say it's more distributed because Blockchains can be infinitesimal amount of nodes, but it's fairly distributed, it's incredibly fast.

Kris Jenkins: (32:00)

Yeah, especially compared to Blockchain it's insanely fast, right?

Fotios Filacouris: (32:04)

Yeah, yeah. So those are kind of some of the basic use cases. Then we do see other fraud related use cases. So for example, if you have a centralized exchange or maybe you're a FinTech company that's providing some kind of Bitcoin trading app or something like that, maybe you're not even a centralized exchange, you're a gateway to centralized exchanges or things like that, you want to do very basic things like analytics on top of the price action of specific crypto assets, right?

Fotios Filacouris: (32:46)

So if I have crypto assets that I want to get similarly to how I would on an index fund or an ETF, I want to see the high/low of the last month, I want to provide analytics on the delta between the two, I want to see the last price of something, that's obviously important as well to display to a web portal or to a mobile app. You see Kafka, you see Kafka streams, you see SQL in those types of use cases, right? Highly volatile changing data sets that you want to do real time analytics on within a window of time. I mean, you just described great use cases for Kafka streams in case SQL, right?

Kris Jenkins: (33:38)

Yeah, absolutely. So this kind of sounds like it's filling exactly the same real time aggregation and notification thing that you'd see if a bank were using mainframe Oracle, they didn't have easy access to modify, right? So one use of Kafka is exactly the same as if your database wasn't Blockchain.

Fotios Filacouris: (34:05)

Yeah, and it's an interesting question, I think the part of the puzzle that we're missing a little bit is that these traditional Blockchain related crypto companies are more cloud native from a cultural perspective. So they do look to technologies that do tend to be a little bit more cloud facing or cloud native types of technologies. So you see a lot of these architectures that are in the cloud. If they're using a database, they're using a managed service database, if they're using any type of event streaming layer, they're using it directly within the cloud fully managed. So I think not only is the technology choices different from where they're deployed, I think that you can draw a very hard line culturally between a bank and how Blockchain companies even develop technology.

Kris Jenkins: (35:08)

I can believe that. Yeah, yeah. You need at least 50 departments to really behave like a bank, right? 50 departments, maybe two of which talk together well. No, that's mean, I'm going to take that back, but the banks I work for.

Fotios Filacouris: (35:22)

It's not mean.

Kris Jenkins: (35:23)

Banks where they're just like little war fiefdoms, right? You can't get that in a startup.

Fotios Filacouris: (35:28)

I think if you talk to a lot of my friends in banking, they would agree maybe not publicly on a podcast, but I'll do it for them, I'll do it for them.

Kris Jenkins: (35:39)

But as well as that, there's this notion of off chain processing, which feels more inherently tied to Blockchain technologies, right? Take me through that in Kafka.

Fotios Filacouris: (35:53)

Yeah, so there's products out there that are called oracles, which is a very-

Kris Jenkins: (36:01)

Not to be confused with Oracle, the company, right?

Fotios Filacouris: (36:06)

Oracle the company, or that fine lady from The Matrix, but there are Oracles out there and what oracles do is they try and marry together via an API, right? So they have their own network and they try and marry together via API and an STK framework a methodology to piece together off chain and on chain data, right? This isn't only useful in transactional types of payment use cases with Blockchain, but it will also be useful in the future when IOT related Blockchain use cases do become popular or do become more usable. And the reason why is because the same reasons why you would want to get this referential data together with on chain data is so that you can make meaningful insights to that data, right?

Fotios Filacouris: (37:08)

So for example, if I'm an oracle and you call out to me, maybe you want to know arbitrage opportunities, what is the price of this coin pairing in another exchange, right? I want to pull, if you're within your Blockchain, your own ecosystem, you have no idea what's outside of it so you need to make calls to this. So where Kafka comes into play is in these types of opportunities with oracles, because we can serve a multitude of use cases in that underlying technology stack that oracles are built on. So again, they are similarly distributed systems, they have governance, tokens and all that stuff, but they are more centralized in the facet that they control their own network, their own Blockchain, right?

Fotios Filacouris: (38:08)

So where does Kafka come into play? Kafka can come into play to scrape metadata associated to the current status and the health of that network so you can do real time reporting on it since the stability similarly to how you would with any type of distributed system that you owned or your company owned, you would want to ensure that it didn't go down, right? So that's the health and reactive type of monitoring of that type of network is a fairly interesting use case.

Fotios Filacouris: (38:53)

The other things that Kafka can do is you can write within a producer a scraping algorithm for an Ethereum smart contract, right? So as data is being consumed back and forth between the Blockchain, you can't actually change the transaction that occurred within Ethereum, but you may want to report what that transaction was outside of that Blockchain to, let's say, a presentation layer on the front end or something like that. That's where Kaka can come into play as well, and where we're seeing it being widely adopted, right? There are some NFT based organizations, there are some Oracles that we know are heavily leveraging Kafka to do that real time kind of status update associated to a smart contract that's been initiated somewhere else. So the off chain and on chain marriage Kafka is well suited to be a facilitator of the off chain stuff.

Kris Jenkins: (40:04)

For data shipping that you don't want to do directly on the chain? Yeah.

Fotios Filacouris: (40:09)

Sure.

Kris Jenkins: (40:09)

I'm not sure if this is related, but I wanted to try and get it clear in my head so maybe you can tell me if we're on the same track. I'm just thinking of something like I want to trade this token for that token on the chain. So I submit my transaction saying I would like to buy 2000 ABCs for 1000 DEFs, right? I'm going to submit that to the chain, now we've got 10 nodes all competing to be the person that mines that, there are now 11 opinions of what the exchange rate is between ABCs, DEFs. Is that where an oracle would come in and how does that play out?

Fotios Filacouris: (40:50)

Spot on 100%, that's where an oracle comes into play and where certain oracles aren't being leveraged within a Blockchain, you do have that divergence that is very painful in distributed systems. When nodes don't agree on things, things go wrong, right? Yeah, so you're spot on there. And I think that is a major example of why oracles are one of the most widely adopted technologies when you are building Blockchains.

Kris Jenkins: (41:26)

Right, okay. And I'm just guessing here, but you could have a system where when I submit that trade, I say, for instance, I think we should use the exchange rate on this Kafka topic at offset 1050. And that way everyone can agree what value immutable value I want to use on that price feed topic, and as long as we agree that's fair, we'd all be looking at the same piece of reference data.

Fotios Filacouris: (41:53)

Yeah, yeah I think it could be used that way. I don't know if it currently is being used that way, right? But I think if we put our solutioning hats on, that could absolutely be... I mean, the way that I would view something like that would be looking at potentially a K table, right? So if I wanted to keep the most up to date price associated to an underlying asset, I knew every time that I hit that topic for that type of request I could broadcast that existing price to my other validation nodes within the cluster and we all agree at the same time that we're using that price.

Kris Jenkins: (42:30)

Yeah, yeah. Does it also play into the idea of... Because we've talked about Kafka being kind of an outward notification system off the Blockchain, does it also play into we've got to notify say a smart contract that something has happened from the outside world? Is it two way?

Fotios Filacouris: (42:50)

So it could be, it could be. A lot of smart contract if you're familiar with the underlying, I mean, you can write a smart contract in JavaScript, I mean, solidity smart contracts, some of them are built in go lang and things like that. So the same way you can embed a producer or consumer logic in any type of code programmatically, you could a smart contract within your producer consumer logic as well, right? So I think it could absolutely be leveraged that way, I just haven't seen it yet. It would be interesting to kind of see how that would work because there's no connector for that, let's just put it that way, Kris. So we'd have to do some custom development, but I think that could be a next step. I'm just hoping we survive this market to see it.

Kris Jenkins: (43:59)

Yeah, it's funny because there's a delay between us recording these podcasts and them being broadcast so whoever's listening to this knows some of the answer at least, some of the outcome of this. Hello, people of the future.

Fotios Filacouris: (44:14)

Throw us a lifeline people.

Kris Jenkins: (44:15)

Yeah, if only we get two way topics that would send data backwards in time, then we could make some serious money.

Fotios Filacouris: (44:22)

Absolutely.

Kris Jenkins: (44:24)

Well, God, where does that leave us? Give me a prediction for the future. I'm going to force your hand. We don't know where the future's going, but give me a prediction. What do you think's going to play out next?

Fotios Filacouris: (44:35)

Okay. SO do we say what the date is or is that something we leave intentionally?

Kris Jenkins: (44:42)

Let's just say that towards the start of Q2, there was an implosion that we are now aware of.

Fotios Filacouris: (44:49)

Yeah, yeah. So I predict that we will go into a bear cycle. So economically, if you're asking from that perspective, I think there's going to be a significant downturn just like there always is every four years in Blockchain crypto related markets. And then you wait for the Bitcoin halvening, the halvening process and markets turn up and things go bull again.

Fotios Filacouris: (45:22)

When you ask me about the technology, I think that is what is going to exponentially increase over the course of the past, let's say, four year cycle, right? The reason why is that you are seeing valuations of some of these Blockchain related companies come in at astronomical numbers, in the billions of dollars. So if I'm a VC, why would I stop investing or speculating on these types of companies, right? And what comes with that is innovation. We're going to see proof of stake technology implemented on Blockchains within the next year, year and a half, which is going to drive the overall price to transact within Blockchains significantly down. And I think once we see that cost go down in the transactionality of leveraging Blockchains, we'll see more mass adoption, right?

Kris Jenkins: (46:28)

If it becomes more practical to make transactions?

Fotios Filacouris: (46:30)

Yeah. Yeah. I mean crypto and Blockchain and all of this stuff has been touted as the every man's system, come and join the network, it will be great and beautiful and everything is sunny. And then you get into you try and transact and buy an NFT on Ethereum and you spend $3,000 just in the mining fee and you realize what kind of common man can do that, right? Not many.

Kris Jenkins: (47:02)

Yeah. I have relatives that can barely find last week's email so we're not there yet, mentioning no relatives in the Jenkins family specifically. I'm going to give you my personal prediction because I worked for Blockchain company for a while and you mentioned JavaScript, they were working on a smart contracts language written in Haskell. So I'm going to say that Haskell is part of the future of Blockchain and that's close to my heart.

Fotios Filacouris: (47:33)

All right, I'm going to need to read up more on Haskell.

Kris Jenkins: (47:40)

On that note, I should probably let you go. Fotios, thank you very much for joining us. It's been an interesting survey of the Blockchain world. Maybe we'll bring you back on in six months and we can see how your predictions did.

Fotios Filacouris: (47:53)

That would be great. Like I said, Kris, I do talk Kafka and Blockchain crypto all the time so it's such an honor to come on the show and talk with you about it and would love to come back.

Kris Jenkins: (48:06)

It's a pleasure, and tell people where they can find your podcast on Blockchain.

Fotios Filacouris: (48:10)

Oh, absolutely. And maybe we can throw it in the show notes, but I have a crypto podcast, a Blockchain podcast that's called New Kids on the Bloq, I kid you not.

Kris Jenkins: (48:22)

Loving the nineties pun.

Fotios Filacouris: (48:26)

And all one word where the block is spelled B-L-O-Q. And you can find me on LinkedIn and the Twitter sphere if you dare enter that space.

Kris Jenkins: (48:42)

Are you on TikTok yet?

Fotios Filacouris: (48:46)

I am a consumer of news on TikTok, but not a poster of things, but maybe I'll be hip to it one of these days.

Kris Jenkins: (48:55)

Well, if you do between now and publication date, we'll put a link to that in the show notes too.

Fotios Filacouris: (48:59)

Awesome, thank you.

Kris Jenkins: (49:00)

Thanks, Foti, good to talk to you.

Fotios Filacouris: (49:01)

Absolutely Cheers.

Kris Jenkins: (49:03)

Well, thank you, Foti. I hope whatever's happening in the market today your portfolio is diverse and stable. That whole discussion reminds me that one of the big changes in software in the past 15 years or so is that our definition of database has only got broader and broader. We include relational databases, sure, but also document databases, key value stores, Blockchains, and of course Kafka. And the trick as always is in choosing the right tool for the job but the trick behind the trick is knowing more about which tools are available. You can't make the right choice without the right information. So I hope you found that all useful.

Kris Jenkins: (49:46)

And whether it's a database or a data store, debate that, if you want to learn more about Kafka, where it fits and how to use it well, check out developer.confluent.io, which is our documentation and tutorial site. It's got everything from code snippets to step by step courses that will help you make a success of your event driven systems. And when you need to get Kafka running quickly, easily, and reliably, then head to Confluent Cloud, which is our fully managed Apache Kafka service. You can get started in minutes and if you use the code PODCAST100 on the billing page, you'll get $100 of extra free credit to run with. And with that, it just remains for me to thank Fotios Filacouris for joining us and you for listening. I've been your host, Kris Jenkins, and I will catch you next time.