Episode description

Coming out of a whirlwind year for the event streaming world, Tim Berglund sits down with Gwen Shapira (Engineering Leader, Confluent), Ben Stopford (Senior Director, Office of the CTO, Confluent), and Michael Noll (Principal Technologist, Office of the CTO, Confluent) to take a guess at what 2021 will bring. 

The experts share what they believe will happen for analytics, frameworks, multi-cloud services, stream processing, and other topics important to the event streaming space. These Apache Kafka® related predictions include the future of the Kafka cluster partitions and removing restrictions that users have found in the past, such as too many variations and excessive concurrency as it relates to your number of partitions.

Ben also thinks that ZooKeeper will continue to maintain open source servers for highly reliable application distribution. Kafka clusters will still be able to keep the most important data while growing in size at record speed with ZooKeeper, although it will no longer be required with KIP-500 removing ZooKeeper dependency. This upgrade allows Kafka and ZooKeeper to run independently in deployment while Kafka’s cluster capability will increase.

Michael expects a continued need for COVID-19 tracking as well as enhanced event streaming capabilities. Ben believes that scalable Tiered Storage for Kafka will increase productivity and benefit workloads. Gwen predicts that databases will become more conventional by the end of next year, leading to new data architectural design with the support of Kafka.

EPISODE LINKS

• KIP-500: Apache Kafka Without ZooKeeper ft. Colin McCabe and Jason Gustafson
• How to set up podcasts on Alexa
• Better to Be Wrong Than Vague: Apache Kafka and Data Architecture Predictions for 2021
• Join the Confluent Community Slack
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Live demo: Kafka streaming in 10 minutes on Confluent Cloud
• Use 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)