Security Pitfalls in Data Engineering: Auth, Secrets, Supply Chain, and Safe Defaults

[0:00]Alam: Welcome back to the Data Engineering Practices Podcast, where we go beyond pipelines and SQL to talk about the real-world challenges facing data teams today. I’m your host, Mark, and this episode is all about security pitfalls in data engineering apps—think authentication, secrets, supply chain, and those seemingly harmless default settings. We’ve got a lot to unpack, and I’m thrilled to have Priya Malhotra here with us. Priya, welcome!

[0:35]Priya Malhotra: Thanks, Mark! I’m excited to dig into this. It’s a topic close to my heart—and honestly, one that doesn’t get enough attention in most data engineering circles.

[0:50]Alam: Absolutely. Let’s start with the big question: Why does security so often get sidelined in data engineering projects? What’s your take?

[1:05]Priya Malhotra: I think it’s a mix of things. Data engineering is usually all about speed—building fast, shipping pipelines, unlocking analytics. Security is seen as someone else’s job, or as something that will slow the team down. But because we’re moving and transforming so much sensitive data, the stakes are actually huge.

[1:40]Alam: That’s a great point. And it feels like data engineering sits at this intersection: you’ve got infrastructure, application code, lots of automation, and sensitive data all in one place.

[2:00]Priya Malhotra: Exactly. And that intersection is where things can go wrong in surprising ways. The attack surface is big. A small mistake—like a misconfigured service account or an exposed environment variable—can open the door to a major incident.

[2:30]Alam: Let’s get practical. Priya, can you share a little about your background and what you see in the field?

[2:45]Priya Malhotra: Sure! I lead the data security team at Datastream Analytics, and my job is to help teams design and operate secure data platforms. I’ve worked with everything from classic ETL systems to cloud-native streaming pipelines, and I’ve seen all kinds of security mishaps—some minor, some… well, let’s just say they became expensive lessons.

[4:00]Alam: I think those lessons are what people tune in for! So, from your vantage point, what makes data engineering apps uniquely vulnerable compared to, say, web apps or mobile apps?

[4:20]Priya Malhotra: One big difference is the number of automated processes. Data engineering relies heavily on service accounts, background jobs, schedulers—machines talking to other machines. Those identities need permissions, and it’s easy to give them too much. The other thing is, data engineers often use a lot of third-party packages and tools, sometimes without thoroughly vetting them.

[5:00]Alam: So, it’s not just about users logging in—it’s about all these non-human actors. Let’s pause and define that. What do you mean by service accounts, and why are they risky?

[5:20]Priya Malhotra: A service account is a digital identity used by applications or automated scripts to access other resources—like databases, storage buckets, or APIs. The risk is they often get broad permissions for convenience. If one of those credentials leaks, anyone who gets hold of it can impersonate that service and do a lot of damage.

[6:40]Alam: That sounds a lot like role creep—where permissions just keep accumulating over time. Is that something you see a lot?

[6:55]Priya Malhotra: All the time! Over time, teams grant more permissions because something breaks or a new feature is added. Suddenly, a service account that started with read-only access can now write to production tables, delete resources, or even spin up new infrastructure.

[7:25]Alam: Let’s make this real. Can you share an example where authentication pitfalls led to a major issue?

[7:40]Priya Malhotra: Sure. There was a case where a data pipeline used a single service account for development and production—because it was easier. The credentials were accidentally committed to a public repository. Within hours, someone used them to access sensitive financial data. The team only noticed when data volumes spiked unexpectedly.

[8:25]Alam: Oof, that’s rough. And it’s not even a sophisticated attack—it’s just a simple mistake.

[8:35]Priya Malhotra: It really is. In that case, the lack of separation between environments, and the absence of monitoring on the service account, made it worse.

[9:15]Alam: Let’s dig into the difference between user authentication and machine authentication. What are the subtleties people miss?

[9:35]Priya Malhotra: User authentication is about humans logging in—think passwords, multi-factor, maybe SSO. Machine authentication is about programmatic access—API keys, certificates, or tokens. People often set stricter rules for users but neglect the machines, which can be far more powerful and persistent.

[10:10]Alam: So, a machine identity can live for months or years, quietly accumulating permissions.

[10:20]Priya Malhotra: Exactly. And if it’s compromised, attackers can move laterally, access logs, exfiltrate data, or even plant malware in the pipeline.

[11:30]Alam: Let’s get into a mini case study here. Can you walk us through a real scenario where a service credential leak led to a production incident?

[11:45]Priya Malhotra: Absolutely. In one project, a junior engineer accidentally left a database password in a configuration file that was checked into a shared repo. No one noticed for weeks. Eventually, a penetration tester found the secret and demonstrated that with just that password, they could download millions of records from the data warehouse.

[12:20]Alam: Wow. How did that get missed in code review?

[12:35]Priya Malhotra: It’s easy to miss, especially if reviewers are looking for logic errors, not secrets. Plus, if there’s no automated scanning for secrets, things slip through.

[14:00]Alam: So, let’s talk about secrets management. What’s secrets sprawl, and why is it so dangerous?

[14:20]Priya Malhotra: Secrets sprawl is what happens when sensitive credentials—like API keys, passwords, tokens—are scattered everywhere: code, config files, CI pipelines, shared documents. The more places a secret lives, the harder it is to track and rotate. That increases the chance of exposure.

[15:00]Alam: I’ve seen teams hardcode credentials directly in source files, especially in notebooks or quick scripts. How does that happen in practice?

[15:20]Priya Malhotra: It usually starts as a shortcut. Someone is debugging locally, they put the password in the file to get things working. Then they forget to remove it—or worse, it gets checked into version control and replicated everywhere.

[15:50]Alam: Let’s be honest, we’ve all done it at some point. What are some ways to avoid secrets sprawl?

[16:05]Priya Malhotra: The key is to use dedicated secrets management tools—like cloud key vaults or environment variable managers—and to automate access as much as possible. There should be clear policies: never commit secrets to code, rotate them regularly, and audit where they’re used.

[16:40]Alam: What about teams that say, 'It’s just internal, only engineers have access'?

[16:55]Priya Malhotra: That’s a dangerous mindset. Insider threats are real, and so are accidents. Plus, repositories have a way of sticking around—old forks, backups, forgotten branches. Once a secret is out, you lose control over it.

[17:00]Alam: Let’s talk about how to implement a proper secret storage solution. What should a team look for?

[17:20]Priya Malhotra: Look for automated rotation, access control, audit logs, and integration with your CI/CD pipeline. It should be easy for developers to fetch secrets securely, without having to know or handle them directly.

[18:00]Alam: What are some trade-offs? I’ve heard some teams say these systems add friction.

[18:15]Priya Malhotra: There’s always a balance. Some tools are harder to set up, or have learning curves. But the friction is worth it. Every shortcut increases risk, and the cost of a breach is way higher than the cost of a little extra setup.

[19:45]Alam: Let’s pivot to defaults. Why are unsafe defaults such a persistent problem in data tools?

[20:00]Priya Malhotra: Because defaults are what most people use! Data engineering tools want to be easy to adopt, so they often ship with permissive settings. That means things like public access enabled, wide-open roles, or logging in the clear.

[20:30]Alam: I’ve seen cloud storage buckets left open to the internet because the default policy was 'public'.

[20:45]Priya Malhotra: Exactly. Unless you take the time to lock things down, you’re vulnerable. And many teams assume 'if it works out of the box, it must be safe.' That’s rarely true.

[22:00]Alam: Let’s do another mini case study. Can you tell us about an incident involving unsafe defaults?

[22:15]Priya Malhotra: Sure. There was a team that set up a cloud storage bucket to store processed data. They didn’t realize that the bucket was configured to allow public reads by default. A security researcher found sensitive records, and the company had to notify affected customers and regulators.

[22:50]Alam: That’s everyone’s nightmare. And it’s so easy to miss with all the moving parts in a modern data stack.

[23:00]Priya Malhotra: Right. Especially with infrastructure as code, a single line can make something public without you realizing. Reviews and automated checks are crucial.

[24:30]Alam: We’ve talked about authentication, secrets, and defaults. Let’s get into supply chain risk. How do dependencies become a security problem in data engineering?

[24:50]Priya Malhotra: Dependency sprawl is real. Data pipelines are built on layers of open source libraries and tools. If any one of those has a vulnerability, it can compromise your entire pipeline. The problem is, most teams don’t audit dependencies regularly.

[25:20]Alam: Is it fair to say the explosion of open source tools has made this risk even greater?

[25:35]Priya Malhotra: Definitely. It’s never been easier to spin up a pipeline using packages from all over the world, but you have to trust hundreds of maintainers you’ve never met. Supply chain attacks are increasingly targeting data engineering workflows.

[26:00]Alam: Can you give an example of a dependency-related incident?

[26:15]Priya Malhotra: I worked with a team using a popular ETL library. The maintainers accidentally published an update that pulled in a vulnerable package. Within days, attackers exploited it to gain access to internal data. The team had no monitoring for dependency changes, so they only found out after it was too late.

[27:10]Alam: That’s a perfect segue to our break. When we come back, we’ll talk about how to audit dependencies and enforce safe defaults in your data pipelines. If you’re listening live, send us your questions—we’ll tackle a few after the break.

[27:25]Priya Malhotra: Looking forward to it!

[27:30]Alam: Alright, welcome back! We’ve covered so much ground on security pitfalls in data engineering so far, but I want to dive even deeper. Let’s pick up where we left off—secrets management in production environments. Can you share a story or two about what happens if teams don’t treat secrets with enough care?

[27:54]Priya Malhotra: Absolutely. One memorable case involved a team that stored their database credentials directly in their ETL scripts, checked into version control. A new engineer cloned the repo, and within days, those credentials were accidentally pushed to a public fork. Luckily, someone noticed, but not before there were signs of suspicious connections to their database.

[28:17]Alam: Ouch. That’s one of those heart-stopping moments for any team. What’s the lesson here, beyond the obvious 'don’t put secrets in code?'

[28:37]Priya Malhotra: The big takeaway is that secrets should be managed outside of source code. Use environment variables, or better yet, dedicated secrets managers like HashiCorp Vault or AWS Secrets Manager. And always assume credentials will eventually leak—so rotate them regularly and monitor for usage anomalies.

[29:00]Alam: I'm glad you said 'assume they’ll leak.' I think that’s a mindset shift for a lot of folks. Now, what about supply chain risks? That’s a phrase we hear a lot, but what does it actually look like for data engineering teams?

[29:27]Priya Malhotra: Great question. In data engineering, your supply chain is mostly your dependencies—libraries, containers, even data connectors. There was a case where a popular open source ETL library was compromised, and the attackers injected malicious code that exfiltrated environment variables at runtime. Teams that didn’t pin versions or vet dependencies were hit hardest.

[29:50]Alam: So what can teams do to protect themselves? Is it practical to audit every library you use?

[30:13]Priya Malhotra: It’s not realistic to audit every line, but you can minimize exposure. Pin your dependency versions, use tools that scan for known vulnerabilities, and prefer well-maintained libraries. Also, pay attention to transitive dependencies—sometimes the problem is buried several layers deep.

[30:33]Alam: That’s a good segue to another point: safe defaults. Can you explain what we mean by that in the data engineering context?

[30:54]Priya Malhotra: Sure. Safe defaults are about making the secure choice the easiest one. For example, configuring new data pipelines to use encrypted connections by default, or denying public access to cloud storage buckets unless explicitly required. Too often, defaults are wide open for the sake of convenience.

[31:17]Alam: Do you see any particular tools or platforms getting this right—or wrong—when it comes to security defaults?

[31:39]Priya Malhotra: Some platforms have improved. For instance, cloud providers now often warn you or block public buckets by default. But many orchestration tools still leave credentials in plain text or skip TLS if you don’t configure it. It’s always worth reviewing the defaults after setup.

[32:00]Alam: Let’s bring this to life with another mini case study. Got one where a safe default—or lack of it—made all the difference?

[32:21]Priya Malhotra: Definitely. A team set up a new data lake, and by default, the storage was publicly accessible. They didn’t realize until someone stumbled on their data via a search engine. Had the default been private, this never would have happened. It was a wake-up call for them and led to a full audit of all their cloud resources.

[32:44]Alam: That’s such a common pitfall. I’ve seen similar stories, unfortunately. Now, shifting gears: authentication and authorization. What are the unique challenges there for data apps?

[33:09]Priya Malhotra: With data apps, you often have machine users—services talking to services. Traditional user management doesn’t always apply. Granting overly broad permissions is common. For instance, a data ingestion job might get full admin access to a warehouse when it only needs insert privileges.

[33:29]Alam: Is that mostly a convenience decision, or do people underestimate the risks?

[33:44]Priya Malhotra: Both. It’s easier to give broad access and move on. But in practice, if those credentials leak, an attacker can do far more damage. Principle of least privilege is huge here—give only what’s needed and nothing more.

[34:01]Alam: Can you share a quick example where this principle saved the day—or where ignoring it led to problems?

[34:21]Priya Malhotra: Sure. There was a pipeline that only needed read access to a reporting database. Someone accidentally gave it write permissions, and a bug in the code wiped out a table. If they’d scoped permissions correctly, the worst case would have been a failed read, not lost data.

[34:42]Alam: That's a tough lesson. So, how do you recommend teams approach privilege management for data engineering workloads?

[35:05]Priya Malhotra: Automate as much as possible. Use infrastructure-as-code to define roles and permissions, so you can review and audit changes. And periodically review those permissions—just because something needed access six months ago doesn’t mean it still does.

[35:26]Alam: Let’s switch to incident response. When things go wrong, what’s the first thing a data team should do if they suspect a security breach?

[35:48]Priya Malhotra: Containment is key. First, revoke any potentially compromised credentials. Then, assess the blast radius: what systems, data, and users were affected? Don’t rush to clean up before you understand what happened—sometimes the root cause is deeper than it looks.

[36:08]Alam: In your experience, are most teams prepared for that kind of incident?

[36:25]Priya Malhotra: Honestly, many aren’t. There’s often no runbook, no pre-arranged contacts, and uncertainty about who owns what. Even just having a basic incident checklist and communication plan can make a huge difference.

[36:44]Alam: Let’s dive into monitoring for a second. What should data teams be logging and monitoring to catch security issues early?

[37:06]Priya Malhotra: You want to log authentication attempts—both successes and failures—changes to permissions, and any access to sensitive data. Monitor for anomalies, like a service account accessing much more data than usual, or logins from unexpected locations.

[37:25]Alam: That brings up an interesting point: in cloud environments, a lot of these logs are available, but do you find that teams actually look at them?

[37:43]Priya Malhotra: Not as often as they should. Logs can be overwhelming, and without alerting and dashboards, it’s easy to miss signals. Setting up simple alerts—like for failed logins or privilege changes—can go a long way.

[38:03]Alam: Let’s do a quick rapid-fire round. I’ll ask you some yes/no or short-answer questions about common security decisions in data engineering. Ready?

[38:09]Priya Malhotra: Let’s do it!

[38:12]Alam: Should secrets ever be passed as command-line arguments?

[38:15]Priya Malhotra: No—those can show up in process listings.

[38:18]Alam: Are hard-coded API keys ever acceptable?

[38:21]Priya Malhotra: Never. Use environment variables or a secrets manager.

[38:24]Alam: Is it safe to use the latest version of every library as soon as it's released?

[38:27]Priya Malhotra: No. Test first, pin versions, and watch for security advisories.

[38:30]Alam: Should you share credentials between multiple pipelines if it’s easier?

[38:34]Priya Malhotra: No. Unique credentials for each pipeline improves traceability and containment.

[38:37]Alam: Is it okay to skip TLS in internal networks?

[38:40]Priya Malhotra: Not recommended. Internal traffic can be intercepted too.

[38:43]Alam: Should you alert on every failed login attempt?

[38:46]Priya Malhotra: Alert on patterns, not every single failure—otherwise you'll get alert fatigue.

[38:49]Alam: Is rotating secrets quarterly enough?

[38:52]Priya Malhotra: It’s a start, but automate more frequent rotation where possible.

[38:56]Alam: Awesome. Thanks for playing along! Now, let’s circle back to real-world mistakes. Can you share another anonymized case where a small oversight led to big trouble?

[39:22]Priya Malhotra: Sure. A team set up a data pipeline to process sensitive customer data. They used a staging bucket for temporary files, but forgot to enable encryption. Months later, during an audit, they discovered that anyone with the bucket name could download those files. It was a simple config miss, but the impact could have been huge if someone outside the company found it.

[39:45]Alam: That’s a classic example of why audits matter. How often should teams do security reviews of their projects?

[40:01]Priya Malhotra: Ideally, at every major release or change. But at a minimum, schedule periodic reviews—maybe quarterly. And don’t forget to review after onboarding new services or dependencies.

[40:20]Alam: Let’s talk about culture for a minute. Do you think security is seen as a blocker in data engineering, or is that changing?

[40:38]Priya Malhotra: It’s improving, but old habits die hard. Security is often seen as extra work, especially under time pressure. But more teams are realizing that a small investment up front saves a lot of pain later.

[40:55]Alam: Are there any strategies that help bring security into the workflow without slowing teams down?

[41:13]Priya Malhotra: Integrate security checks into CI/CD pipelines—lint for secrets, scan dependencies, enforce code reviews for security-sensitive changes. If it’s part of the build process, it becomes routine rather than a separate hurdle.

[41:31]Alam: That’s a great point. Now, if you had to pick one overlooked security risk in data engineering, what would it be?

[41:51]Priya Malhotra: I’d say inter-service communication—those internal APIs or message queues. They’re often assumed to be safe because they’re 'internal,' but if someone gets into your network, they can move laterally fast.

[42:05]Alam: How can teams mitigate that?

[42:18]Priya Malhotra: Enforce authentication and authorization everywhere, even on internal services. Use mutual TLS if possible, and audit service-to-service permissions regularly.

[42:32]Alam: Let’s move to our final mini case study. Can you walk us through a scenario where a supply chain attack threatened a data platform?

[42:56]Priya Malhotra: Of course. One team relied on a third-party connector for a major cloud data warehouse. A new version was released by a rogue maintainer, who added code to siphon secrets to an external host. Automated dependency updates pulled it in. They only caught it because their outbound traffic monitoring flagged a suspicious domain. It was a close call.

[43:18]Alam: Wow. That’s a sobering reminder. So, outbound traffic monitoring—would you recommend that for all data teams?

[43:33]Priya Malhotra: Definitely. It can alert you to exfiltration attempts you’d otherwise miss. Even just flagging new destinations or unexpected traffic volumes can help you catch issues early.

[43:51]Alam: Alright, as we get close to wrapping up, I want to make this super practical. Could you walk us through an implementation checklist for securing data engineering apps? Imagine you’re advising a team starting from scratch.

[44:08]Priya Malhotra: Absolutely. Here’s a conversational checklist:

[44:13]Priya Malhotra: First, inventory all data flows—know what moves where, and who or what accesses each system.

[44:20]Priya Malhotra: Second, externalize secrets—use a secrets manager, and never store credentials in code or config files.

[44:27]Priya Malhotra: Third, enforce least privilege on everything—machine users, pipelines, and human users alike.

[44:33]Priya Malhotra: Fourth, review and pin all dependencies, and set up automated scanning for vulnerabilities.

[44:39]Priya Malhotra: Fifth, configure safe defaults—encrypted storage, private buckets, and secure connections by default.

[44:45]Priya Malhotra: Sixth, set up monitoring and alerting on key events—especially access to sensitive data and permission changes.

[44:52]Priya Malhotra: Seventh, run periodic security reviews and incident response drills, so the team’s ready when something goes wrong.

[44:58]Alam: That’s a solid list. For listeners who want to take action tomorrow, which two steps would you prioritize first?

[45:09]Priya Malhotra: I’d say move secrets out of code immediately, and review your permissions—those are the lowest-hanging fruit with the biggest risk reduction.

[45:20]Alam: Let’s take a quick detour into data privacy. How do privacy regulations intersect with the security practices we’ve been talking about?

[45:37]Priya Malhotra: They’re tightly linked. Data privacy rules require you to protect personal data, limit access, and respond quickly to breaches. The security fundamentals we’ve discussed—like access controls and audit trails—are often required by these regulations.

[45:50]Alam: Are there any unique privacy pitfalls you’ve seen in real-world data pipelines?

[46:08]Priya Malhotra: Absolutely. One team accidentally logged sensitive user data to a shared log file that many engineers could access. It was meant for debugging, but it created a privacy exposure. Always sanitize logs and limit who can view them.

[46:23]Alam: That’s a great reminder. As we wrap up, I’d love to hear your final word on building a security-first mindset in data engineering.

[46:41]Priya Malhotra: Security isn’t a one-time project. It’s a habit and a culture. Encourage everyone on the team to ask 'what if this leaked?' or 'what if this was misused?' before shipping anything new. And make it easy for people to do the right thing by default.

[46:58]Alam: Amazing advice. Before we close, let's recap the key takeaways. I’ll start—first, never store secrets in code or config files. What’s next?

[47:07]Priya Malhotra: Second, follow least privilege—only grant what’s truly needed.

[47:14]Alam: Third, monitor and log everything critical, especially around access and data movement.

[47:21]Priya Malhotra: Fourth, review your dependencies and default configurations regularly.

[47:28]Alam: Fifth, make incident response and security reviews a regular part of your process.

[47:35]Priya Malhotra: And finally, foster a culture where security is everyone’s job—not just the security team’s.

[47:41]Alam: Perfect. Any resources you’d recommend for teams looking to go deeper on this topic?

[47:56]Priya Malhotra: I’d suggest looking into the OWASP Top Ten, especially their guidance for APIs and cloud apps. Also, most major cloud providers have security best practices docs tailored for data workloads—those are gold.

[48:10]Alam: Fantastic. Well, I think that’s a great note to end on. Thank you so much for joining us and sharing so many practical insights.

[48:18]Priya Malhotra: Thanks for having me! Always a pleasure to talk shop and help teams build safer data systems.

[48:27]Alam: Alright, before we let you go, let’s do a super-quick recap checklist for listeners. I’ll call these out, and you just say 'yes' or 'no' if it’s a must-do. Ready?

[48:29]Priya Malhotra: Ready!

[48:31]Alam: Secrets in code?

[48:32]Priya Malhotra: No!

[48:34]Alam: Default public storage buckets?

[48:35]Priya Malhotra: No!

[48:37]Alam: Enforce least privilege?

[48:38]Priya Malhotra: Yes!

[48:40]Alam: Dependency scanning in CI/CD?

[48:41]Priya Malhotra: Yes!

[48:43]Alam: Automated alerting on suspicious access?

[48:44]Priya Malhotra: Yes!

[48:46]Alam: Periodic security reviews?

[48:47]Priya Malhotra: Yes!

[48:52]Alam: Awesome. That’s the checklist. For everyone listening, these are non-negotiables if you want to avoid the most common security pitfalls in your data engineering projects.

[49:00]Priya Malhotra: And if you do just these, you’re ahead of a lot of teams out there.

[49:07]Alam: Before we say goodbye, any last words of encouragement for teams that feel overwhelmed by all this?

[49:21]Priya Malhotra: Start small. Pick one risk area—maybe secrets or permissions—and focus on that this month. Improvements add up, and it’s never too late to level up your security posture.

[49:32]Alam: Love that. Focus on progress, not perfection. Thank you again for being here and for sharing all this wisdom.

[49:40]Priya Malhotra: Thanks for the great questions and for shining a light on these important issues.

[49:54]Alam: And to everyone listening, thanks for joining us on this deep dive into data engineering security. Be sure to subscribe for more practical conversations like this. Until next time, stay safe and keep building smarter.

[50:00]Priya Malhotra: Take care, everyone!

[50:15]Alam: (Outro music fades in) You’ve been listening to the Softaims podcast, where we help you navigate the complex world of modern data engineering. If you found this episode helpful, please share it with a colleague and leave us a review. See you on the next episode!

[50:30]Priya Malhotra: (Outro music continues, fades out)

[55:00]Alam: Episode complete. Thanks for listening!