Episode description

On the podcast this week, your hosts Stephanie Wong and Mark Mirchandani talk about the data processing tool Apache Beam with guests Pablo Estrada and Kenneth Knowles.

Kenn starts us off with an overview of how Apache Beam began and how Cloud Dataflow was involved. The unique batch and stream method and emphasis on correctness garnered support from developers early on and continues to attract users. Pablo helps us understand why Beam is a better option for certain projects looking to process large amounts of data. Our guests describe how Beam may be a better fit than microservices that could become obsolete as company needs change.

Next, we step back and take a look at why batch and stream is the gold standard of data processing because of its balance between low latency and ease of “being done” with data collection. Beam’s focus on the correctness of data and correctness in processing that data is a core component. With good data, processing becomes easier, more reliable, and cheaper. Kenn gives examples of how things can go wrong with bad data processing. Beam strives for the perfect combination of low latency, correct data, and affordability. Users can choose where to run Beam pipelines, from other Apache software offerings to Dataflow, which means excellent flexibility. Our guests talk about the pros and cons of some of these options and we hear examples of how companies are using Beam along with supporting software to solve data processing challenges.

To get started with Beam, check out Beam College or attend Beam Summit 2022.

Kenn Knowles is chair of the Apache Beam Project Management Committee. Kenn has been working on Google Cloud Dataflow—Google’s Beam backend—since 2014. Kenn holds a PhD in programming languages from the University of California, Santa Cruz.

Pablo is a Software Engineer at Google, and a management committee member for Apache Beam. Pablo is big into working on an open source project, and has worked all across the Apache Beam stack.

• Under the sea: Building the world’s fiber optic internet video
• • Discovering Data Centers videos
• Google Data Cloud Summit site
• It’s official—Google Distributed Cloud Edge is generally available blog
• • GCP Podcast Episode 228: Fastly with Tyler McMullen podcast
• Save big by temporarily suspending unneeded Compute Engine VMs—now GA blog

• Apache Beam site
• Apache Beam Documentation site
• Dataflow site
• Apache Flink site
• Apache Spark site
• Apache Samza site
• Apache Nemo site
• Spanner site
• BigQuery site
• Beam College site
• Beam College on Github site
• Beam Developer Mailing List email
• Beam User Mailing List email
• Beam Summit site

Mark is working on a new Apache Beam video series Getting Started Wtih Apache Beam

Stephanie Wong and Mark Mirchandani