CI/CD Security Pitfalls: Auth, Secrets, Supply Chain, and Building Safer Defaults

[0:00]Pavel: Welcome back to the show. Today, we’re diving into one of the most overlooked but critical aspects of modern software delivery: security pitfalls in CI/CD applications. We’re talking authentication, secrets, supply chain risks, and why the defaults in your pipeline might be working against you.

[0:40]Pavel: I’m your host, Jamie, and joining me in the studio is Morgan Li, Lead Security Engineer at BuildOps Secure. Morgan, thanks for being here!

[1:00]Morgan Li: Thanks for having me, Jamie. I’m excited to dig into this topic. It’s one of those things that sounds niche, but touches every team shipping code today.

[1:20]Pavel: Absolutely. Let’s kick off with some context. Why is CI/CD such a hotbed for security issues right now? What’s changed compared to, say, older deployment practices?

[2:15]Morgan Li: Great question. The shift to continuous integration and continuous delivery means code is moving faster than ever from dev environments to production. That speed is amazing for innovation, but it also means that the blast radius for mistakes or attacks is much bigger. One leaked secret or misconfigured permission can move from a side branch to production in minutes.

[3:00]Pavel: So the very thing that makes CI/CD powerful—automation and speed—also makes it risky?

[3:15]Morgan Li: Exactly. You’ve got automated systems with access to code, infrastructure, and sensitive data. If you’re not careful with how they’re set up and secured, you’re basically handing attackers the keys to the kingdom.

[3:35]Pavel: Let’s get to know you a bit, Morgan. What’s your background? How did you end up specializing in CI/CD security?

[4:50]Morgan Li: I started as a DevOps engineer, actually. I spent a lot of time building pipelines, automating deployments, and patching things together with scripts. But over time, I realized how easy it was to accidentally create security holes—especially with secrets and permissions. I moved into security to help teams build things right the first time, instead of cleaning up after leaks.

[5:30]Pavel: Was there a particular incident that made it click for you?

[5:40]Morgan Li: There was, actually. Early on, I saw a team push an environment variable containing a database password into their shared CI logs. It stayed there for months, and nobody noticed until someone outside the team stumbled onto it. That was an eye-opener.

[6:30]Pavel: That’s a nightmare scenario. Let’s pause and define some terms. When we say 'authentication' and 'authorization' in CI/CD, what do we actually mean?

[7:00]Morgan Li: Sure. Authentication is about verifying who—or what—someone is. In CI/CD, that could be a developer, a service account, or an automated process. Authorization is about what that entity is allowed to do—like which repositories a build process can access, or which cloud resources a deployment can modify.

[7:45]Pavel: How do those concepts break down in a typical pipeline? Where do the most common mistakes happen?

[8:20]Morgan Li: The classic mistake is using overly broad permissions. For example, a build runner might get credentials that grant admin access to every environment. That’s convenient, but if those credentials leak, an attacker can do anything. Another is not segmenting access—for instance, letting every pipeline job access production secrets, regardless of what it’s actually doing.

[9:30]Pavel: So, it’s about scope and boundaries. We hear a lot about OAuth tokens and API keys being leaked. How does that actually happen in practice?

[9:50]Morgan Li: It’s more common than people think. Sometimes a developer hardcodes a token in a script, or forgets to add a secret to the right place and ends up committing it to version control. In other cases, tokens get echoed in logs, or accidentally exposed via debug output during a build. CI/CD logs are a goldmine for attackers if not tightly controlled.

[10:40]Pavel: Are there warning signs that a pipeline is at higher risk for these kinds of leaks?

[11:00]Morgan Li: Yes—if your pipeline lets anyone view logs, or if you don’t have secrets scanning in place, you’re at risk. Another red flag is having too many secrets managed as environment variables with loose controls. And if you’re using third-party integrations and haven’t reviewed their permissions, that’s another blind spot.

[12:10]Pavel: Let’s dig into secrets management. Where do secrets actually live in most pipelines?

[12:30]Morgan Li: Ideally, secrets should be stored in a dedicated secrets manager or vault, integrated with your CI/CD tool. But in reality, they’re everywhere—environment variables, config files, sometimes even in plain text inside scripts or code. The sprawl is real.

[13:20]Pavel: So, if someone is just getting started, what’s the easiest mistake to make with secrets?

[13:40]Morgan Li: Honestly, it’s convenience. Teams want things to work, so they’ll put secrets where the pipeline can easily reach them, even if it’s insecure. That might mean embedding them in YAML files or storing them in project-level environment variables with broad access.

[14:45]Pavel: Let’s get concrete. Could you walk us through a real incident where secrets leaked through a CI/CD pipeline?

[15:10]Morgan Li: Definitely. One team I worked with was using a managed CI platform. A developer accidentally pushed an API key into a public repository. The pipeline picked up the repo, ran a build, and printed environment variables—including the key—into the build logs. Those logs were publicly accessible for hours. A bot found the key and started making fraudulent API calls before anyone noticed.

[16:00]Pavel: Brutal. How did they handle cleanup?

[16:20]Morgan Li: It was a scramble. They had to rotate the key, audit all usage, and add scanning to catch future leaks. But the real fix was tightening log access and stopping secrets from being echoed in the first place.

[17:30]Pavel: Let’s talk about secrets scanning. A lot of tools promise to catch leaks. How effective are they, really?

[17:55]Morgan Li: They’re useful, but not a silver bullet. Scanners can miss secrets that are obfuscated, split across variables, or use non-standard formats. Plus, if a secret leaks to logs or a third-party system before the scanner runs, it’s already too late.

[18:40]Pavel: So, defense in depth—don’t just rely on scanning. What else should teams be doing?

[19:00]Morgan Li: Exactly. Limit who can view logs, use short-lived credentials, and integrate secrets managers directly with your pipeline. And design your build steps so that secrets are only available when absolutely necessary.

[19:50]Pavel: Let’s shift to supply chain risks. What does that mean in the context of CI/CD?

[20:15]Morgan Li: Supply chain risk is all about the dependencies and third-party tools your pipeline pulls in. If you’re building with open source packages or using external plugins, a compromise in one of those can give attackers a foothold inside your build process.

[20:55]Pavel: How does that play out in the real world? Any cautionary tales?

[21:20]Morgan Li: Plenty. I saw a case where a popular package used in a build script was hijacked by an attacker who published a malicious update. The build picked up the new version automatically, and suddenly all downstream deployments had a hidden backdoor.

[22:00]Pavel: Let’s pause and recap for listeners: if your pipeline just grabs dependencies without pinning versions or verifying sources, you’re at risk for this kind of attack.

[22:20]Morgan Li: Exactly. Always pin your dependencies, use checksums, and consider tools that verify the provenance of packages before your pipeline uses them.

[22:50]Pavel: Mini case study time. Can you give us a quick story about a supply chain compromise that caught a team off guard?

[23:10]Morgan Li: Sure. A fintech startup used a plugin from a public marketplace to speed up their deploys. Unbeknownst to them, the plugin maintainer sold the project, and the new owner slipped in code that exfiltrated secrets during builds. The team only found out after unusual traffic showed up in their logs.

[24:00]Pavel: That’s scary. How can teams defend against marketplace plugin risks like that?

[24:20]Morgan Li: Vet your plugins and extensions, stick to officially supported ones, and monitor for suspicious behavior. If possible, run third-party code in isolated environments with minimal privileges.

[24:50]Pavel: Let’s talk about defaults. Out-of-the-box, how safe are most CI/CD platforms?

[25:10]Morgan Li: Honestly? Not very. Defaults are chosen for ease of setup, not security. Permissions are often broad, secrets may be stored in plaintext, and logging is verbose. If you don’t tweak the settings, you’re probably leaving doors open.

[25:50]Pavel: What’s the first thing you change when onboarding a new CI/CD tool?

[26:10]Morgan Li: I lock down permissions—make sure jobs only have the access they need. Then I configure secret storage to use a vault, and turn off excessive logging or at least restrict log access. Finally, I review all default integrations and disable anything we’re not actively using.

[26:50]Pavel: Let’s squeeze in a quick disagreement. Some folks argue that tightening defaults slows developers down. What’s your take?

[27:10]Morgan Li: I get it—security can feel like friction. But in my experience, the small up-front cost saves massive headaches later. That said, it’s important to work with developers and make sure controls are practical, not just restrictive.

[27:30]Pavel: That’s a great point. Security for its own sake isn’t helpful if it brings everything to a halt. Let’s recap what we’ve covered so far: authentication boundaries, secrets sprawl, supply chain dangers, and why safe defaults matter.

[27:30]Pavel: Alright, picking up where we left off—so we've talked about the basics of authentication and secrets, but let's dig deeper. What are some of the more subtle or overlooked pitfalls you've seen when it comes to secret management in CI/CD pipelines?

[27:54]Morgan Li: One thing that pops up repeatedly is secrets inadvertently getting baked into artifacts or logs. For example, a developer adds a debug statement, and suddenly an API key is printed in the pipeline logs. Those logs get stored or shipped to a third-party service, and now you've got a pretty serious risk.

[28:11]Pavel: Yeah, and once it's in the logs, it's basically out there forever unless you realize and scrub it. Any memorable stories?

[28:31]Morgan Li: Absolutely. I remember a team that used a popular cloud-based CI service. Their deployment script echoed environment variables for debugging, not realizing one was a production database password. Years later, they were breached, and the investigation traced it back to old logs with that password. They’d rotated some secrets, but not that one.

[28:49]Pavel: Oof. That’s tough. So, what’s the best way to avoid that kind of mistake?

[29:07]Morgan Li: First, treat logs as sensitive by default. Mask secrets in logs, and use your CI/CD platform’s secret masking features. Also, minimize what gets logged during builds—don’t log environment variables unless absolutely necessary.

[29:20]Pavel: And I guess regular secret rotation helps, too?

[29:32]Morgan Li: Definitely. Secret rotation limits the blast radius if something’s exposed. And automation helps—tools that rotate secrets can plug into your pipeline so you’re not relying on manual updates.

[29:47]Pavel: Let’s talk about supply chain risks. Nowadays, teams use tons of open-source dependencies and containers. Where do things go wrong?

[30:07]Morgan Li: A big one is blindly trusting dependencies or container images. It’s easy to pull a base image or a library and not realize it’s been compromised or contains vulnerabilities. Attackers target popular packages for this reason.

[30:22]Pavel: So what’s the first line of defense?

[30:37]Morgan Li: Implement a policy: only allow images and dependencies from trusted sources. Use image signing and verification, and lock down dependency versions with hashes. And always scan dependencies as part of your pipeline.

[30:52]Pavel: Can you share a quick case study—maybe around a supply chain mishap?

[31:13]Morgan Li: Sure. There was a fintech company using a community-maintained Docker image for their app. One day, a new maintainer pushed a malicious update that included a crypto miner. Their pipeline pulled the latest image automatically. No alarms went off, but their cloud bill skyrocketed. Only after a week did they notice and trace it back.

[31:28]Pavel: Wow, so the lesson there is what—never use 'latest' tags?

[31:39]Morgan Li: Exactly. Always pin to specific versions and review changes in dependencies and images before updating. Automate scanning, but also add human review for anything critical.

[31:52]Pavel: Let’s shift to the concept of safe defaults. What does that mean in the context of CI/CD security?

[32:10]Morgan Li: Safe defaults mean configuring your CI/CD systems so that, if someone makes a mistake or skips a step, things don’t fall into an insecure state by default. For example, restricting pipeline permissions, disabling shell access, or requiring explicit approval for deployments.

[32:25]Pavel: Are there any defaults you see that are especially risky?

[32:41]Morgan Li: Yes—open permissions are a huge one. Some CI/CD platforms grant very broad permissions by default, like letting any pipeline job deploy to production or access sensitive secrets. That’s dangerous if someone misconfigures a job or if a malicious pull request slips through.

[32:56]Pavel: So, should teams go for principle of least privilege everywhere?

[33:07]Morgan Li: Always. Start with the least privilege possible and only escalate when absolutely necessary. Make it hard to accidentally give out more access than needed.

[33:18]Pavel: That reminds me—what about third-party integrations? Teams love to plug in code quality tools, notifications, all sorts of bots.

[33:34]Morgan Li: That’s another big risk area. Every integration you add is a new attack surface. Some tools request read/write access to your repo or secrets. If those tools are compromised, you’re exposed. Vet integrations carefully and grant the minimum required permissions.

[33:46]Pavel: Let’s get practical: What’s your approach to reviewing a new CI/CD tool before rolling it out?

[34:05]Morgan Li: First, read the docs to understand what permissions and secrets the tool needs. Check community reputation, look for security whitepapers, and see how often it’s updated. Then test it in a sandbox environment—never in prod first.

[34:19]Pavel: Alright, I want to do a quick rapid-fire round. I’ll throw out some scenarios, and you give me your first instinct answer. Ready?

[34:22]Morgan Li: Let’s do it.

[34:26]Pavel: Number one: Environment variables or config files for secrets?

[34:33]Morgan Li: Secrets manager, but if you must, environment variables—never in source-controlled config files.

[34:39]Pavel: Second: Self-hosted runners versus cloud runners?

[34:47]Morgan Li: Cloud runners for convenience and patching, but self-hosted if you need total control—just watch for stale hosts.

[34:53]Pavel: Third: Allowing pipeline jobs to run on pull requests from forks?

[34:59]Morgan Li: Not with secrets enabled! Always restrict secret access for forked PRs.

[35:04]Pavel: Four: Dependency pinning—critical or nice-to-have?

[35:07]Morgan Li: Absolutely critical.

[35:10]Pavel: Five: Open-source CI/CD tools versus paid platforms?

[35:18]Morgan Li: Depends on your threat model—open source gives transparency, paid platforms offer managed security. Evaluate both.

[35:22]Pavel: Six: Single sign-on for CI/CD tools—worth it?

[35:27]Morgan Li: Absolutely. Centralized access control and audit trails.

[35:32]Pavel: Alright, last one: Disabling shell access by default in build jobs?

[35:36]Morgan Li: Yes, unless there’s a clear, documented need.

[35:42]Pavel: Awesome. Thanks for playing along. Let’s circle back—what about human error? Even with all these controls, mistakes happen.

[35:58]Morgan Li: That’s where layered defenses come in—don’t rely on any one control. Regular audits, automated policy enforcement, and peer reviews catch mistakes early. Training is also huge—make sure everyone understands the risks and safe practices.

[36:10]Pavel: What’s your approach to security reviews in the CI/CD context?

[36:27]Morgan Li: Start with a threat modeling session—map out who can access what, where secrets flow, what could go wrong. Then, run through the pipeline step by step and look for weak spots. After that, automate checks—linting for secrets, dependency scans, permissions audits.

[36:40]Pavel: Can you walk us through an anonymized example where a missed detail nearly caused a big problem?

[37:02]Morgan Li: Sure. A SaaS company had a staging pipeline that reused production credentials for a quick integration test. They assumed the pipeline couldn’t leak secrets. Then, a contractor with limited access ran a test job with extra logging, and the secrets were exposed in logs accessible to the whole team. Luckily, someone spotted it within hours, but it was a close call.

[37:16]Pavel: That’s a classic staging-env pitfall. Any advice for isolating environments?

[37:31]Morgan Li: Never share secrets or credentials between environments. Use separate credentials, restrict access, and monitor usage. Ideally, even test data should be synthetic so there’s no risk of leaking sensitive info.

[37:44]Pavel: How do you see teams balancing speed and security? CI/CD is all about fast feedback, but security checks can feel like friction.

[38:00]Morgan Li: The key is to automate as much as possible and make security checks developer-friendly. Fast static analysis, incremental scans, and clear feedback help. And bake security in early—don’t bolt it on later.

[38:14]Pavel: Makes sense. Let’s talk about onboarding new team members. What’s essential for getting them up to speed on CI/CD security?

[38:28]Morgan Li: Documentation is critical—clear runbooks and checklists. Walk them through the key risks and the controls in place. And have them shadow a more experienced engineer for their first few deployments.

[38:43]Pavel: Alright, time for another mini case study. Can you share an example where a seemingly small misconfiguration in CI/CD led to a wider incident?

[39:06]Morgan Li: Definitely. A media company had a build job that uploaded artifacts to a public S3 bucket for convenience. One day, a sensitive internal tool got built and published there by mistake. A search engine indexed it, and a researcher found it within days. No real breach, but it could have been much worse.

[39:19]Pavel: That’s a nightmare. How do you avoid those accidental exposures?

[39:32]Morgan Li: Automate checks for public buckets, set up access controls and monitoring on storage. And always review which jobs can publish to what locations. Least privilege applies here too.

[39:45]Pavel: Let’s talk about incident response. If you discover a secret leak or suspicious activity in your pipeline, what’s the first thing to do?

[40:00]Morgan Li: Immediately revoke and rotate the affected secrets. Then, audit logs to see if there’s evidence of misuse. Communicate with stakeholders and follow your incident response plan. The faster you act, the less damage.

[40:11]Pavel: And what about longer-term remediation?

[40:23]Morgan Li: Root cause analysis. Figure out what allowed the leak—was it a config mistake, missing controls, or a process gap? Fix it, update documentation, and retrain the team if needed.

[40:34]Pavel: Are there any tools or frameworks you recommend for ongoing CI/CD security monitoring?

[40:50]Morgan Li: There are some great open source options—tools that scan for secrets in code, dependency checkers, and policy enforcement engines. Some CI/CD platforms have plugins for this. Also, consider integrating with your SIEM for broader monitoring.

[41:03]Pavel: Let’s shift gears to compliance. For teams in regulated industries, what are the extra gotchas in CI/CD?

[41:19]Morgan Li: You have to be extra careful with audit trails—every deployment, every change must be logged and attributable. Also, policies around data residency, retention, and access controls are critical. Sometimes you’ll need manual approvals in the pipeline.

[41:32]Pavel: Do you recommend any special training for teams in those environments?

[41:45]Morgan Li: Yes—compliance training tailored to your sector, plus specific guidance on secure development and deployment practices. Make sure everyone understands the legal and reputational risks.

[41:58]Pavel: Alright, as we approach our wrap-up, I want to get your take on the biggest mindset shifts teams need to make around CI/CD security.

[42:16]Morgan Li: Teams need to see security as an enabler for speed, not a blocker. It’s about building trust into your pipeline so you can deploy faster, not slower. Also, making everyone responsible for security—not just the ops team.

[42:29]Pavel: What about the role of culture—how do you help teams shift from ‘move fast and break things’ to ‘move fast and don’t break security’?

[42:45]Morgan Li: It starts with leadership. Make security visible, celebrate when someone finds a bug before it ships, and share lessons learned from incidents. Normalize asking questions about pipeline security in every code review or standup.

[43:00]Pavel: Let’s do a quick checklist for implementing secure CI/CD pipelines. Can you walk us through a mental runbook?

[43:05]Morgan Li: Absolutely. Here’s a checklist you can follow:

[43:10]Morgan Li: 1. Inventory your CI/CD tools and integrations. Know what you’re running and where.

[43:16]Morgan Li: 2. Lock down permissions—apply least privilege everywhere, especially for deployment and secret access.

[43:23]Morgan Li: 3. Store secrets in a dedicated secrets manager. Never hard-code them or check them into source control.

[43:29]Morgan Li: 4. Scan code, dependencies, and containers for known vulnerabilities as part of every build.

[43:36]Morgan Li: 5. Pin all dependencies and base images to specific, trusted versions. Avoid using 'latest' tags.

[43:42]Morgan Li: 6. Enable audit logging and monitor pipeline activity for suspicious behavior.

[43:48]Morgan Li: 7. Regularly review and rotate secrets, and automate the process when possible.

[43:54]Morgan Li: 8. Train your team on secure CI/CD practices and make security a routine part of onboarding.

[44:01]Pavel: That’s a fantastic list. Anything you’d add for folks just starting to modernize their pipeline?

[44:13]Morgan Li: Start small—don’t try to solve everything at once. Pick one or two high-impact improvements, implement them, and iterate. And document as you go.

[44:25]Pavel: Let’s open it up for some final thoughts. What’s the one thing you wish every CI/CD team would do tomorrow?

[44:35]Morgan Li: Review your current secret storage and access. If you’re not using a dedicated secrets manager, make that your next priority.

[44:43]Pavel: And for the more advanced teams—what’s the next level?

[44:53]Morgan Li: Look into supply chain security—signing artifacts, verifying dependencies, and setting up automated policy enforcement for new dependencies.

[45:03]Pavel: Last couple of questions before we sign off. What excites you most about the future of CI/CD security?

[45:16]Morgan Li: The growing adoption of policy as code and automated security gates. Teams can codify their security requirements and have them enforced automatically, reducing the human error factor.

[45:23]Pavel: And what keeps you up at night?

[45:34]Morgan Li: The complexity of CI/CD pipelines—when things get too sprawling, it’s hard to know what’s really going on. Shadow pipelines, forgotten integrations, and privilege creep are real dangers.

[45:42]Pavel: Final actionable tip for listeners?

[45:52]Morgan Li: Schedule regular pipeline reviews. Even once a quarter, just walking through what your pipeline does and who can access what can reveal surprises.

[46:01]Pavel: Great advice. Any resources or tools you’d recommend for folks wanting to learn more?

[46:14]Morgan Li: Check out the documentation from your CI/CD platform, and look into open source security tools like secret scanners and dependency checkers. There are also great online courses and blogs focused on DevSecOps.

[46:26]Pavel: Alright, before we wrap, let’s summarize our main takeaways for listeners.

[46:36]Morgan Li: Sure. First, treat your CI/CD pipeline as critical infrastructure—secure it like you would your production environment.

[46:43]Morgan Li: Second, implement least privilege everywhere—permissions, secrets, integrations.

[46:50]Morgan Li: Third, automate as many security checks as possible so security becomes part of your workflow, not a blocker.

[46:57]Morgan Li: Fourth, stay vigilant—monitor, review, and never assume you’re finished.

[47:06]Pavel: Perfect. Thanks so much for sharing your wisdom and stories today. It’s been super insightful.

[47:13]Morgan Li: Thanks for having me. Always great to talk about making pipelines safer!

[47:20]Pavel: Before we sign off, any shout-outs or final words for our listeners?

[47:29]Morgan Li: Just a reminder—security is everyone’s job. Don’t be afraid to ask questions or suggest improvements, even if you’re new to CI/CD.

[47:39]Pavel: Love it. Thanks again for joining us. For everyone listening, check out the episode notes for links to resources and our implementation checklist.

[47:46]Morgan Li: And feel free to reach out if you have questions or want to share your own CI/CD stories!

[47:54]Pavel: Thanks so much. That wraps up today’s episode of the Softaims podcast. Stay secure, and we’ll see you next time.

[48:00]Morgan Li: Take care, everyone.

[48:07]Pavel: Alright, we’ll close out with our implementation checklist one more time for folks who tuned in late. Here’s what you should do:

[48:14]Pavel: 1. Inventory all CI/CD tools and integrations.

[48:18]Pavel: 2. Apply least privilege to all permissions.

[48:21]Pavel: 3. Use a secrets manager, never hard-coded secrets.

[48:25]Pavel: 4. Scan all code, dependencies, and images.

[48:29]Pavel: 5. Pin dependencies and images—never use 'latest'.

[48:32]Pavel: 6. Enable audit logs and monitor pipeline activity.

[48:36]Pavel: 7. Regularly review and rotate secrets.

[48:39]Pavel: 8. Train your team and keep security front and center.

[48:45]Pavel: Thanks again for joining us. If you found this useful, subscribe and share with your team. Until next time, happy shipping and stay safe out there.

[48:49]Morgan Li: Bye everyone!

[48:54]Pavel: And that’s a wrap. This has been the Softaims podcast—see you in the next episode.

[55:00]Pavel: End of episode.