Python in Production today: From Quick Scripts to Scalable Systems

[0:00]Omar: Welcome back to the Python stack podcast. This is episode two, and today we are staying with Python, but we are changing the question. In episode one, we asked why Python still matters today. Today, we are asking what happens after Python starts mattering too much.

[0:35]Omar: Because that is the strange thing about Python. It often enters a company quietly. Someone writes a script to clean a file. Someone writes a notebook to test a model. Someone writes a small API to expose an internal tool. Someone writes a scheduled job to move data from one place to another. At first, nobody calls it architecture. Nobody calls it infrastructure. Nobody calls it a platform.

[1:20]Omar: Then six months later, that little script is running every hour. The notebook logic is feeding a dashboard. The small API is used by three teams. The scheduled job is part of billing. And suddenly, the thing that started as quick Python has become production Python.

[2:05]Omar: That transition is what we are talking about today. Not Python as a toy. Not Python as only a learning language. Not Python as only notebooks. We are talking about Python as a serious production stack: services, workers, data pipelines, AI systems, automation, deployment, monitoring, and security.

[3:00]Omar: To help us walk through that, I am joined by Marcus Bennett, a senior Python platform engineer at Northstar Data Systems. Marcus has worked on Python backend platforms, AI workflow systems, data pipelines, and internal developer tooling. Marcus, welcome.

[3:28]Marcus Bennett: Thanks for having me. I like this topic because Python has a reputation for being easy, and that reputation is both true and misleading. Easy to start does not mean easy to operate. Easy to read does not mean easy to scale. Easy to write does not mean easy to maintain.

[4:05]Omar: That is probably the whole episode in three sentences.

[4:12]Marcus Bennett: It really is. Python gives you momentum. That is why people love it. You can turn an idea into working code very quickly. But production systems punish unmanaged momentum. If you never introduce structure, tests, deployment discipline, observability, and ownership, Python can become a pile of successful accidents.

[5:00]Omar: A pile of successful accidents is painfully accurate.

[5:08]Marcus Bennett: And to be clear, that is not a Python-only problem. Every popular language has messy systems. But Python is especially good at helping people move from idea to code, so it also creates more chances for prototype code to survive longer than it should.

[6:00]Omar: So today we are not asking whether Python is good. We are asking how to be good at Python when the stakes are real.

[6:12]Marcus Bennett: Exactly. Python is good. The ecosystem is good. The community is good. The tooling is better than it used to be. The runtime is evolving. But none of that removes the need for engineering judgment.

[6:30]Omar: Let us start with Python’s real power. People usually say Python is popular because it is simple. Is that the whole story?

[6:42]Marcus Bennett: No. Simplicity is part of it, but the deeper power is connection. Python connects people, systems, and ideas. It connects beginners to programming. It connects researchers to production teams. It connects APIs to data pipelines. It connects AI models to applications. It connects cloud services, databases, command-line tools, notebooks, queues, and dashboards.

[7:35]Omar: So Python is not just a language. It is organizational glue.

[7:42]Marcus Bennett: Yes, and that glue role is a big reason Python keeps winning. A machine learning engineer can write Python. A backend engineer can write Python. A data analyst can write Python. A DevOps engineer can write Python. A security engineer can write Python. A finance analyst might write Python. That shared language reduces handoff friction.

[8:35]Omar: But that also means a lot of people with different standards are writing code in the same language.

[8:43]Marcus Bennett: That is the tradeoff. Python democratizes software creation. That is good. But production systems need more than access. They need consistency. If five different teams write Python in five completely different styles, with five dependency tools, five logging patterns, five deployment methods, and five testing standards, the company eventually pays for that.

[9:40]Omar: So production Python starts with shared expectations.

[9:45]Marcus Bennett: Yes. Not heavy bureaucracy. Not giant architecture documents nobody reads. Just clear defaults. This is how we create projects. This is how we manage dependencies. This is how we format code. This is how we test. This is how we log. This is how we deploy. This is how we handle secrets. This is how we decide whether a notebook becomes a service.

[10:30]Omar: Let us talk about the runtime. Python 3.14 is a big deal. Free-threaded Python is officially supported. There are binary releases for experimental JIT work. There are language changes around annotations and template strings. What should production teams actually care about?

[11:00]Marcus Bennett: They should care, but they should not panic or overreact. Python 3.14 is important because it signals where CPython is going. Free-threading being officially supported is a major milestone. The experimental JIT work is also part of a broader performance story. But production teams should treat these as platform evolution, not immediate magic.

[11:50]Omar: So nobody should wake up tomorrow and say, we are free-threaded now, all performance problems are solved.

[11:58]Marcus Bennett: Absolutely not. That would be a misunderstanding. Free-threaded Python can allow Python threads to run in parallel without the traditional GIL limitation, but whether that helps your application depends on your workload, dependencies, native extensions, memory behavior, and thread safety. It is not a universal speed button.

[12:55]Omar: Give me an example.

[13:00]Marcus Bennett: Imagine a service that spends most of its time waiting on Postgres and calling external APIs. Free-threading may not change the most important bottleneck. Better indexes, better connection pooling, better timeouts, better caching, and fewer unnecessary calls might matter more. But imagine a CPU-heavy Python workload that can safely split work across threads and whose dependencies behave well in a free-threaded build. That could become more interesting.

[14:00]Omar: So the correct first step is measurement.

[14:05]Marcus Bennett: Always. Measure before and after. Benchmark real workloads. Check memory usage. Check latency distribution. Check throughput. Check error rates. Check whether your dependencies support the runtime mode you want. The production mindset is: interesting feature, controlled experiment, evidence, then adoption.

[15:00]Omar: What about the JIT work?

[15:05]Marcus Bennett: Same rule. It is exciting. It shows serious investment in making Python faster. But if you are running a production platform, you do not make promises based on theoretical speed. You test your actual code. Maybe it helps. Maybe it does not. Maybe your biggest bottleneck is still database time. Maybe your hot path is already inside NumPy or PyTorch or a Rust extension. The only honest answer comes from profiling.

[16:00]Omar: Now tooling. Modern Python feels different from old Python. A lot of teams are using pyproject.toml, uv, Ruff, type checkers, and cleaner CI workflows. What changed?

[16:20]Marcus Bennett: The ecosystem got tired of slow, fragmented workflows. For a long time, Python packaging felt like a maze: pip, virtualenv, setup.py, requirements files, Poetry, Pipenv, Conda, build backends, lockfiles, and lots of team-specific rituals. Some of that complexity still exists, but the direction is better. Modern tools are trying to make Python projects faster, clearer, and more reproducible.

[17:20]Omar: Where does uv fit into that?

[17:25]Marcus Bennett: uv is important because it combines speed with project management. It can manage dependencies, environments, scripts, lockfiles, and Python versions in a way that feels much faster than many older workflows. For production teams, speed matters because developers run tools more often when they are not painful.

[18:15]Omar: That sounds small, but it changes behavior.

[18:20]Marcus Bennett: Exactly. If installing dependencies takes forever, people avoid fresh installs. If linting is slow, people wait for CI. If tests are hard to run, people do not run them locally. If the project setup is unclear, onboarding becomes tribal knowledge. Good tooling reduces friction around good habits.

[19:15]Omar: And Ruff?

[19:20]Marcus Bennett: Ruff changed the conversation around linting and formatting because it is very fast and covers a lot of functionality. It can replace many separate tools in many projects. That consolidation matters. Teams do not need to debate ten style tools when one tool can handle most of the everyday work.

[20:05]Omar: So the point is not that every team must use the exact same tools. The point is to have a coherent toolchain.

[20:12]Marcus Bennett: Correct. Use uv, Poetry, pip-tools, Conda, or whatever fits your environment. Use Ruff or another style setup. Use mypy, pyright, basedpyright, or another type checker if that is your choice. The production question is: can a developer clone the repo, install dependencies, run tests, format code, type check, and start the service without guessing?

[21:10]Omar: That is a very practical definition of maturity.

[21:15]Marcus Bennett: Yes. Mature tooling is not about being trendy. It is about making the correct workflow boring.

[22:00]Omar: Let us talk about project structure. Python projects can get messy quickly. What does a maintainable Python project look like?

[22:15]Marcus Bennett: A maintainable project has clear boundaries. It is obvious where the application starts. It is obvious where configuration lives. It is obvious where domain logic lives. It is obvious where infrastructure code lives. It is obvious where tests are. It is obvious how dependencies flow. The main problem in messy Python projects is not syntax. It is hidden coupling.

[23:05]Omar: Hidden coupling meaning everything imports everything?

[23:10]Marcus Bennett: Yes. Route handlers importing database clients directly. Business logic reading environment variables. Utility modules with side effects. Model code opening files at import time. Configuration scattered across modules. Test fixtures that secretly depend on global state. It all works until someone tries to change something.

[24:00]Omar: What is the better pattern?

[24:05]Marcus Bennett: Keep the domain logic as clean as possible. Push IO to the edges. Make dependencies explicit. Load configuration once and pass it where needed. Keep route handlers thin. Keep workers focused. Avoid doing real work at import time. Make tests able to run without accidentally connecting to production services. These are boring rules, but they prevent real pain.

[25:00]Omar: Python makes imports feel easy, but import-time behavior can become dangerous.

[25:08]Marcus Bennett: Very dangerous. Import-time database connections, network calls, reading large files, loading models, registering global state, starting background tasks — those things make applications harder to test, slower to start, and harder to reason about. A production Python app should be deliberate about startup.

[26:00]Omar: What about configuration?

[26:05]Marcus Bennett: Configuration should be typed, validated, and environment-aware. Do not let every module call os.environ randomly. Have a configuration object. Validate required settings at startup. Separate secrets from normal configuration. Make local development easy without making production unsafe.

[27:00]Omar: That sounds like the difference between code that runs and code that can be operated.

[27:08]Marcus Bennett: Exactly. Production code has to be operated by humans under pressure. Clear structure is not academic. It is incident response preparation.

[28:00]Omar: Now backend frameworks. Python gives us Django, FastAPI, Flask, and other choices. How should teams choose?

[28:15]Marcus Bennett: Choose based on the shape of the product, not fashion. Django is excellent when you need a full application stack: ORM, migrations, admin, authentication patterns, forms, security defaults, and a big ecosystem. FastAPI is strong for typed APIs, OpenAPI generation, async-friendly service design, and modern API teams. Flask remains useful when you want a lightweight, flexible application with minimal framework assumptions.

[29:20]Omar: So there is no single best Python web framework.

[29:25]Marcus Bennett: Correct. There is a best fit for a context. A small webhook receiver does not need the same framework as a multi-tenant admin-heavy SaaS app. An internal ML inference API does not have the same needs as a content management platform. Framework decisions should follow product needs, team experience, security needs, and operational constraints.

[30:30]Omar: And backend is not only request-response.

[30:35]Marcus Bennett: Exactly. Many Python systems do their most important work outside HTTP. Workers process jobs. Schedulers trigger tasks. Queues absorb spikes. Pipelines transform data. Consumers read from streams. Batch jobs reconcile state. A web framework may be only the front door.

[31:30]Omar: What is a common mistake with background jobs?

[31:35]Marcus Bennett: Treating them as a trash can for complexity. People say, just move it to a background job, as if that solves everything. But background jobs need idempotency, retries, dead-letter handling, visibility, timeout rules, concurrency limits, and ownership. A job that fails silently is worse than a request that fails loudly.

[32:40]Omar: What about async Python?

[32:45]Marcus Bennett: Async is valuable when the workload waits on IO. It can help services handle many concurrent operations efficiently. But async is not a magic performance upgrade. If your endpoint does CPU-heavy work, blocks on synchronous libraries, or holds the event loop hostage, async can make the system more fragile, not faster.

[33:40]Omar: So async requires discipline too.

[33:45]Marcus Bennett: Yes. Know which libraries are async. Know where blocking calls happen. Use timeouts. Cancel work properly. Do not mix sync and async casually. Measure event loop delays. And do not choose async because it looks modern. Choose it because the concurrency model fits the workload.

[34:00]Omar: Let us move to AI and data. Python is dominant there. But you often say notebooks are not production systems. What do you mean?

[34:15]Marcus Bennett: A notebook is a thinking environment. It is excellent for exploration, visualization, model experiments, data inspection, and communication. But notebooks often hide state. Cells run out of order. Outputs may not match the current code. Credentials may be pasted into cells. Data assumptions may be implicit. That is fine for exploration, but risky for production.

[35:15]Omar: So what should happen when notebook work becomes important?

[35:20]Marcus Bennett: Extract the logic into modules. Add tests. Define inputs and outputs. Track dependencies. Version data where needed. Add validation. Automate execution. Add logging and metrics. Decide who owns it. A notebook can be the seed, but production needs a plant with roots.

[36:20]Omar: AI apps add even more uncertainty.

[36:25]Marcus Bennett: They do. Traditional software is already hard because inputs vary. AI systems add probabilistic behavior, model drift, prompt sensitivity, retrieval quality, latency variance, cost variance, and safety concerns. If you build AI applications in Python, you need evaluations, monitoring, fallback paths, and clear failure behavior.

[37:30]Omar: What does a good LLM-backed Python service need?

[37:35]Marcus Bennett: It needs request validation, prompt versioning, model configuration management, retrieval observability if using RAG, output validation, safety checks, rate-limit handling, cost tracking, latency tracking, and regression evaluations. You need to know not only whether the code runs, but whether the answer quality is still acceptable.

[38:45]Omar: That is different from ordinary unit testing.

[38:50]Marcus Bennett: Yes. Unit tests still matter, but AI systems need behavioral evaluation. You may test that the pipeline calls the retriever correctly, but you also need to test whether the retrieved context is useful and whether the generated answer follows policy. That requires examples, scoring, review, and monitoring over time.

[40:00]Omar: Let us talk testing more broadly. What does serious Python testing look like?

[40:12]Marcus Bennett: It starts with accepting that tests are not only for correctness. Tests are also documentation, design pressure, refactoring insurance, and production risk reduction. In Python, where dynamic behavior is common, tests are especially important because some mistakes only show up at runtime.

[41:00]Omar: What should teams test?

[41:05]Marcus Bennett: Test business rules, validation, permissions, API contracts, database behavior, serialization, background jobs, retries, idempotency, migrations, configuration loading, and failure paths. Do not only test the happy path. Production rarely respects the happy path.

[42:00]Omar: What is a weak test?

[42:05]Marcus Bennett: A weak test proves the mock returns what the mock was told to return. Or it checks that a function was called without checking the outcome that matters. Or it hardcodes implementation details so every refactor breaks the suite. A good test protects behavior. A weak test protects structure.

[43:00]Omar: And observability?

[43:05]Marcus Bennett: Observability is how you test production while it is alive. You need structured logs, metrics, traces, error reporting, health checks, and useful dashboards. Logs should include request IDs or correlation IDs. Metrics should show rate, errors, duration, saturation, queue depth, worker failures, memory, CPU, and dependency latency.

[44:10]Omar: What do Python teams often miss?

[44:15]Marcus Bennett: They log messages instead of events. A message says something happened. An event gives context: which operation, which tenant, which route, how long it took, what failed, whether it retried, and what the impact was. During incidents, context is everything.

[45:15]Omar: So observability is not just adding a logging library.

[45:20]Marcus Bennett: No. Observability is a design habit. You decide what questions future you will need to answer at 2 a.m. Then you make sure the system emits enough safe, structured information to answer them.

[46:00]Omar: Performance and security. Let us start with performance. What actually makes Python systems slow?

[46:12]Marcus Bennett: Often, not Python itself. Slow database queries. Missing indexes. Too many network calls. Large JSON payloads. Loading too much data into memory. Repeating expensive work. Calling external APIs synchronously. Bad caching. Inefficient serialization. Model inference latency. Cold starts. Contention in workers. Those are usually bigger problems than the language.

[47:10]Omar: So the first optimization is profiling.

[47:15]Marcus Bennett: Yes. Profile the code. Trace the request. Look at database timings. Measure p95 and p99 latency. Check memory usage. Check queue depth. Check CPU. Check external dependencies. Python has optimization options, but choosing the right one requires knowing the bottleneck.

[48:10]Omar: And security?

[48:15]Marcus Bennett: Python security starts with inputs, dependencies, secrets, and execution boundaries. Validate untrusted input. Avoid unsafe deserialization. Protect environment variables. Do not commit credentials. Review packages. Be careful with install scripts and CI permissions. Keep production images small. Separate dev tools from runtime dependencies where possible.

[49:15]Omar: AI-generated code makes this harder.

[49:20]Marcus Bennett: It does, because AI can produce code that looks confident and clean but misses real risks. It may skip validation, swallow exceptions, leak secrets in logs, use an outdated package, write blocking code in async functions, or generate tests that do not test anything meaningful. AI is useful, but it should not lower review standards.

[50:20]Omar: What should code reviewers ask when reviewing Python code, AI-generated or not?

[50:28]Marcus Bennett: What happens on bad input? What happens when the database is down? What happens when the external API times out? Does this leak sensitive data? Does it retry safely? Is it idempotent? Is the dependency necessary? Is this observable? Can we test it? Can the author explain it?

[51:30]Omar: Let us close with a practical checklist. A team is using Python seriously today. What should they do this quarter?

[51:45]Marcus Bennett: First, standardize project creation. Every production Python repo should have a clear pyproject.toml, dependency workflow, lockfile strategy, formatter, linter, type checker decision, test command, and local run command.

[52:20]Marcus Bennett: Second, audit runtime versions. Know which Python versions you run in production, CI, notebooks, containers, and developer machines. Plan upgrades before end-of-life pressure. Test Python 3.14 deliberately if its features matter to your workloads.

[52:55]Marcus Bennett: Third, separate experiments from production. Notebooks and scripts are fine, but once they affect customers, money, compliance, or core operations, they need ownership, tests, deployment discipline, logs, and rollback plans.

[53:30]Marcus Bennett: Fourth, improve observability. Add structured logs, request IDs, traces for important paths, useful metrics, alerting, and dashboards that reflect user impact, not just server activity.

[54:00]Marcus Bennett: Fifth, review dependencies and secrets. Remove unused packages. Update risky ones. Lock what needs locking. Scan but also use judgment. Rotate exposed secrets. Stop storing credentials in notebooks, local files, and random CI variables.

[54:30]Omar: Final sentence: what is production Python today?

[54:36]Marcus Bennett: Production Python today is not just Python that runs. It is Python that can be understood, tested, deployed, observed, secured, and changed without fear.

[54:50]Omar: Marcus Bennett, thanks for joining us.

[54:53]Marcus Bennett: Thanks for having me.

[54:56]Omar: For everyone listening, the takeaway is simple: Python helps you start fast, but production rewards teams that finish carefully.

[55:00]Omar: End.