Episode description
Episode description
SummaryAs data architectures become more elaborate and the number of applications of data increases, it becomes increasingly challenging to locate and access the underlying data. Gravitino was created to provide a single interface to locate and query your data. In this episode Junping Du explains how Gravitino works, the capabilities that it unlocks, and how it fits into your data platform.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Your host is Tobias Macey and today I'm interviewing Junping Du about Gravitino, an open source metadata service for a unified view of all of your schemas
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Gravitino is and the story behind it?
- What problems are you solving with Gravitino?
- What are the methods that teams have relied on in the absence of Gravitino to address those use cases?
- What led to the Hive Metastore being the default for so long?
- What are the opportunities for innovation and new functionality in the metadata service?
- The documentation suggests that Gravitino has overlap with a number of tool categories such as table schema (Hive metastore), metadata repository (Open Metadata), data federation (Trino/Alluxio). What are the capabilities that it can completely replace, and which will require other systems for more comprehensive functionality?
- What are the capabilities that you are explicitly keeping out of scope for Gravitino?
- Can you describe the technical architecture of Gravitino?
- How have the design and scope evolved from when you first started working on it?
- Can you describe how Gravitino integrates into an overall data platform?
- In a typical day, what are the different ways that a data engineer or data analyst might interact with Gravitino?
- One of the features that you highlight is centralized permissions management. Can you describe the access control model that you use for unifying across underlying sources?
- What are the most interesting, innovative, or unexpected ways that you have seen Gravitino used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Gravitino?
- When is Gravitino the wrong choice?
- What do you have planned for the future of Gravitino?
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.
- Gravitino
- Hadoop
- Datastrato
- PyTorch
- Ray
- Data Fabric
- Hive
- Iceberg
- Hive Metastore
- Trino
- OpenMetadata
- Alluxio
- Atlan
- Spark
- Thrift