Building Real-Time Data Pipelines with Microsoft Azure, Databricks, and Confluent

As most developers and architects know, data always needs to be accessible no matter what happens outside of the system. This week, Tim Berglund virtually sits down with Anna McDonald (Principal Customer Success Technical Architect, Confluent) to discuss how Automatic Observer Promotion (AOP) can help solve the Apache Kafka 2.5 datacenter dilemma, a feature now available in Confluent Platform 6.1 and above.

Running operations on the cloud in a scaling organization can be time consuming, error prone, and tedious. This episode addresses manual upgrades and rolling restarts of Confluent Cloud clusters during releases, fixes, experiments, and the like, and more importantly, the progress that’s been made to switch from manual operations to an almost fully automated process. Rashmi Prabhu, a software engineer on the Control Plane team at Confluent, has the opportunity to help govern the data plane that comprises all these clusters and enables API-driven operations on these clusters.

When building solutions for customers in Microsoft Azure, it is not uncommon to come across customers who are deeply entrenched in the Kafka ecosystem and want to continue expanding within it. Thus, figuring out how to connect Azure first-party services to this ecosystem is of the utmost importance. Ryan CrawCour (Engineer, Microsoft) explains how you can use a connector to feed events from your Kafka infrastructure into Azure Cosmos DB, as well as how to get changes from your database system back into their Kafka topics.