Modernizing Legacy Data Systems With Airflow at Procter & Gamble with Adonis Castillo Cordero
Episode description
Legacy architecture and AI workloads pose unique challenges at scale, especially in a global enterprise with complex data systems. In this episode, we explore strategies to proactively monitor and optimize pipelines while minimizing downstream failures.
Adonis Castillo Cordero, Senior Automation Manager at Procter & Gamble, joins us to share actionable best practices for dependency mapping, anomaly detection and architecture simplification using Apache Airflow.
Key Takeaways:
(03:13) Integrating legacy data systems into modern architecture.
(05:51) Designing workflows for real-time data processing.
(07:57) Mapping dependencies early to avoid pipeline failures.
(09:02) Building automated monitoring into orchestration frameworks.
(12:09) Detecting anomalies to prevent performance bottlenecks.
(15:24) Monitoring data quality to catch silent failures.
(17:02) Prioritizing responses based on impact severity.
(18:55) Simplifying dashboards to highlight critical metrics.
Resources Mentioned:
https://www.linkedin.com/in/adoniscc/
Procter & Gamble | LinkedIn
https://www.linkedin.com/company/procter-and-gamble/
Procter & Gamble | Website
https://azure.microsoft.com/en-us/products/monitor/
https://aws.amazon.com/lookout-for-metrics/
https://www.montecarlodata.com/
https://www.astronomer.io/events/roadshow/london/
https://www.astronomer.io/events/roadshow/new-york/
https://www.astronomer.io/events/roadshow/sydney/
https://www.astronomer.io/events/roadshow/san-francisco/
https://www.astronomer.io/events/roadshow/chicago/
Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning