Episode description
Episode description
SummaryIn this episode of the Data Engineering Podcast Rajan Goyal, CEO and co-founder of Datapelago, talks about improving efficiencies in data processing by reimagining system architecture. Rajan explains the shift from hyperconverged to disaggregated and composable infrastructure, highlighting the importance of accelerated computing in modern data centers. He discusses the evolution from proprietary to open, composable stacks, emphasizing the role of open table formats and the need for a universal data processing engine, and outlines Datapelago's strategy to leverage existing frameworks like Spark and Trino while providing accelerated computing benefits.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
- Your host is Tobias Macey and today I'm interviewing Rajan Goyal about how to drastically improve efficiencies in data processing by re-imagining the system architecture
- Introduction
- How did you get involved in the area of data management?
- Can you start by outlining the main factors that contribute to performance challenges in data lake environments?
- The different components of open data processing systems have evolved from different starting points with different objectives. In your experience, how has that un-planned and un-synchronized evolution of the ecosystem hindered the capabilities and adoption of open technologies?
- The introduction of a new cross-cutting capability (e.g. Iceberg) has typically taken a substantial amount of time to gain support across different engines and ecosystems. What do you see as the point of highest leverage to improve the capabilities of the entire stack with the least amount of co-ordination?
- What was the motivating insight that led you to invest in the technology that powers Datapelago?
- Can you describe the system design of Datapelago and how it integrates with existing data engines?
- The growth in the generation and application of unstructured data is a notable shift in the work being done by data teams. What are the areas of overlap in the fundamental nature of data (whether structured, semi-structured, or unstructured) that you are able to exploit to bridge the processing gap?
- What are the most interesting, innovative, or unexpected ways that you have seen Datapelago used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Datapelago?
- When is Datapelago the wrong choice?
- What do you have planned for the future of Datapelago?
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Datapelago
- MIPS Architecture
- ARM Architecture
- AWS Nitro
- Mellanox
- Nvidia
- Von Neumann Architecture
- TPU == Tensor Processing Unit
- FPGA == Field-Programmable Gate Array
- Spark
- Trino
- Iceberg
- Delta Lake
- Hudi
- Apache Gluten
- Intermediate Representation
- Turing Completeness
- LLVM
- Amdahl's Law
- LSTM == Long Short-Term Memory