Data Engineering stack

Data Engineering podcast

Ten long-form episodes per stack: people who shipped real Data Engineering software for messy human problems—inventory, bookings, tours, money, and the spreadsheets that came before. Each page has details, show notes, timestamps, a YouTube link, and a full transcript.

Data Engineering episodes

6 episodes

Real-World Data Engineering Patterns: Boundaries, Testing, and Maintainability

#1: Real-World Data Engineering Patterns: Boundaries, Testing, and Maintainability

Real-World Data Engineering Patterns: Boundaries, Testing, and Maintainability

In this episode, we dive deep into the essential architectural patterns in data engineering that stand the test of real-world teams and production demands. We focus on the often overlooked, gritty details that separate elegant designs from those that actually survive scale, team turnover, and shifting business requirements. You’ll hear practical stories about defining clear boundaries between data domains, keeping pipelines testable and robust, and engineering for maintainability in environments where nothing stays static for long. We’ll break down what goes wrong when teams skip these fundamentals, how to design for both flexibility and reliability, and why the best patterns often emerge from hard-earned lessons. By the end, you’ll have actionable strategies for building data systems that are resilient, testable, and ready for the realities of modern teams.

View Episode
Data Engineering Performance: Profiling, Bottlenecks, and Practical Optimizations

#2: Data Engineering Performance: Profiling, Bottlenecks, and Practical Optimizations

Data Engineering Performance: Profiling, Bottlenecks, and Practical Optimizations

Today’s episode goes beyond the basics to explore how high-performing data engineering teams tackle system slowdowns and maximize throughput. We break down the art and science of profiling data pipelines, uncovering where bottlenecks really hide, and why common assumptions about performance can mislead even experienced engineers. Our guest brings hands-on strategies for practical optimizations—ranging from smarter resource allocation to real-world tuning of batch and streaming jobs. We share stories of cascading failures, unexpected wins, and the subtle trade-offs between speed, cost, and maintainability. Whether you’re wrangling ETL jobs, orchestrating data lakes, or scaling ingestion for analytics, this deep dive will arm you with actionable frameworks to diagnose and elevate your own data systems.

View Episode
Resilient Data Engineering: API Integrations, Idempotency, Rate Limits, and Navigating Real-World Failures

#3: Resilient Data Engineering: API Integrations, Idempotency, Rate Limits, and Navigating Real-World Failures

Resilient Data Engineering: API Integrations, Idempotency, Rate Limits, and Navigating Real-World Failures

In this episode, we unpack the intricate world of designing APIs and integrations specifically for data engineering workflows, focusing on the practical realities of idempotency, rate limiting, and handling failures under real-world constraints. Through hands-on examples and anonymized case studies, we explore why these concepts aren’t just theoretical best practices but essential for reliable pipelines and integrations. Listeners will hear stories of what goes wrong when APIs are misdesigned, how teams recover from common pitfalls, and frameworks for building robust data flows. We clarify key terminology, debate architectural trade-offs, and examine how to balance performance with resilience. The conversation highlights actionable patterns and anti-patterns for teams building and scaling data-intensive systems. Whether you’re a data engineer, API developer, or technical leader, you’ll gain insights you can apply to your current stack.

View Episode
Security Pitfalls in Data Engineering: Auth, Secrets, Supply Chain, and Safe Defaults

#4: Security Pitfalls in Data Engineering: Auth, Secrets, Supply Chain, and Safe Defaults

Security Pitfalls in Data Engineering: Auth, Secrets, Supply Chain, and Safe Defaults

Modern data engineering applications handle vast amounts of sensitive data, but security is often an afterthought until something goes wrong. In this episode, we pull back the curtain on the most common and surprising security pitfalls in data engineering projects: from misconfigured authentication and careless secret management, to overlooked supply chain vulnerabilities and the dangers of unsafe defaults. With real-world examples and practical tips, we help listeners understand not only what can go wrong, but also how to proactively build security into every stage of their data pipelines. Learn how teams get tripped up by cloud misconfigurations, dependency sprawl, and accidental data exposures—and what to do about it. Whether you’re an engineer, architect, or just data-curious, this episode gives you actionable ways to keep your data engineering apps secure by design.

View Episode
Operational Excellence in Data Engineering: Monitoring, Incident Response, and Deployment Discipline

#5: Operational Excellence in Data Engineering: Monitoring, Incident Response, and Deployment Discipline

Operational Excellence in Data Engineering: Monitoring, Incident Response, and Deployment Discipline

What does it take for data engineering teams to achieve true operational excellence in today’s complex data landscape? In this episode, we dive deep into the foundational pillars of robust monitoring, effective incident response, and disciplined deployment practices. Listeners will learn how to build proactive alerting systems, structure playbooks for the unexpected, and foster a culture where deployment reliability is a team sport. Through practical examples and real-world case studies, our guest unpacks common pitfalls and the subtle trade-offs between agility and stability. Whether you’re leading a data platform or contributing to a pipeline, this conversation delivers actionable strategies for making reliability a core part of your day-to-day data engineering work.

View Episode
Futureproof Data Modeling & Migrations: Avoiding Costly Rewrites in Data Engineering

#6: Futureproof Data Modeling & Migrations: Avoiding Costly Rewrites in Data Engineering

Futureproof Data Modeling & Migrations: Avoiding Costly Rewrites in Data Engineering

How can data engineering teams design models and migrations that adapt to change, minimize rework, and keep business moving? In this episode, we tackle the art of futureproof data modeling and the realities of evolving data requirements. Our guest shares war stories from production, explores migration frameworks, and explains how to spot early warning signs of brittle models. Listeners will learn actionable strategies for decoupling schemas, planning for data evolution, and balancing speed with long-term maintainability. Whether you’re designing a new pipeline or inheriting legacy tables, this episode will help you sidestep the most painful pitfalls. Tune in for hands-on advice and real-world examples to keep your data infrastructure nimble.

View Episode

More Episodes by Stack