Velocity is real. The efficiency gains from prompt-driven development are genuine and, used correctly, valuable. But in regulated environments, velocity that bypasses engineering rigour is not a gain; it is a high-interest loan whose repayment terms most organisations have not examined.

I have sat on both sides of the table. Years in software development and then years in product management. From that vantage point, the current enthusiasm for prompt-driven product development looks different to me than it does to most people in the room. The tools are impressive. The prototypes are genuinely impressive. The speed at which a product manager can go from problem statement to something interactive has changed what early-stage validation looks like, and that change is real. When a prototype costs two hours instead of two weeks, teams stop debating whether to test an idea and simply test it. That is a meaningful improvement in how product discovery works.

But, the gap between a working prototype and a production-ready system in a regulated environment is not a gap of polish or features. It is a gap of engineering rigour, governance, and accountability, and in the current climate, that gap is being treated as a detail rather than a risk. We are entering what I would call the era of high-gloss, high-debt development, with software that looks production-ready on day one and becomes a liability by day thirty. As a product manager who has been on the engineering side of that spectrum, I will discuss the risks that solely prompt-driven development carry, and how to ensure a balance.

The Hidden Dangers of ‘Stateless’ AI Software

When a product manager uses prompt-driven tools to build, what they often produce is what I think of as stateless software. Not stateless in the technical sense, but stateless in the sense that the code has no memory of the architectural constraints, compliance requirements, or operational context that make a system safe to run in a regulated environment. That gap shows up in three specific places, and each of them has consequences that are invisible in the demonstration but painful in production.

The first is the absence of observability. Non-functional requirements — including monitoring, logging, and the instrumentation that allows engineers to diagnose failures — are the PM’s responsibility to specify, with engineering’s input, as acceptance criteria in user stories. When software is built without this discipline, outages get reported by customers rather than by the engineering team. Prompt-generated builds, created outside the standard engineering workflow, routinely omit this instrumentation because nobody thought to ask for it and the tool had no reason to include it. In a regulated financial environment, this is not a minor oversight. It is a control gap. A system that cannot provide an audit trail of what it did, when, and to whose data, does not meet the baseline requirements for deployment regardless of how well it performs in a demo. As teams building systems for regulated industries have found, user stories for any automated capability need a set of non-functional requirements covering performance, compliance, security, observability, and operational requirements; not just what the system does, but how reliably, how securely, and with what evidence trail.

The second is operational fragility. Production software in regulated institutions is built within deployment pipelines that manage change, enforce testing, and create audit-traceable releases. In financial systems, even bugs need to be understood before they are fixed. What looks like a bug might actually be edge-case handling that is load-bearing for compliance or business logic. Code generated outside these pipelines, and moved towards production without going through them, bypasses the change management controls that regulators expect to see evidenced. GitClear’s analysis of over 211 million changed lines of code between 2020 and 2024 shows a 60% decline in refactored code, as teams favour feature velocity over codebase health. Technical debt introduced through rapid generation compounds more rapidly and at greater scale than traditional debt, and organisations that focus solely on speed are likely to encounter what amounts to an 18-month wall of compounding liability.

The third is what I call the logic hallucination gap. This is where interface looks right, the user journey works, and the data appears to flow correctly, but the underlying business logic may be probabilistic rather than deterministic, thus producing outputs that look correct in most cases and fail in the edge cases that matter most.

The Developer-PM as the Bridge

What I have said so far is not an argument for slowing down. It is an argument for a specific kind of product management capability: the developer-cum-product-manager. The product manager who has come through engineering carries a predictive mental model that does something prompt-driven tools cannot do, one that anticipates production failure from a specification. Before a line of code is written, before a prototype is demonstrated to a steering committee, a technically grounded PM is already asking the questions that the demo does not answer.

The first role this creates is what I think of as the auditor of intent. Technical leads, architects, and security specialists should add non-functional requirements that cover the system’s performance, operations, and compliance needs. The PM’s role is to ensure those requirements are defined and included from the outset, not retrofitted after the prototype has already shaped expectations. Latency, concurrency, security, audit traceability, error handling, and data integrity under load are not visible in a prototype. All of them will determine whether the system can actually be deployed in a regulated institution. A PM who cannot name and specify these requirements is not fully doing the job regardless of how quickly they can build a prototype. Non-functional requirements protect value as they make functionality usable, safe, compliant, and affordable to operate. The PM is the professional responsible for ensuring that they are present.

The second role is protecting the long-term roadmap. Leadership often takes the demonstration at face value. When a developer shows a working prototype that was built in an afternoon, the natural conclusion is that it is ready to go. The technically grounded PM is the person who has to correct that inference with specificity. Here is what the prototype does not include. Here is what production requires. Here is what the timeline actually looks like when engineering rigour, security review, and compliance validation are included. That input prevents the kind of delivery failure that damages the product, the team, and the institution’s regulatory standing.

As delivery speed increases, product managers face more decision points, not fewer. When prototyping and building become faster, the premium on sound judgement increases. The ability to make good decisions consistently in conditions of uncertainty becomes the irreplaceable core of the role. This is what the developer-PM must possess.

The goal is not to replace engineers with prompting PMs. The goal is to use the PM’s technical depth as a control mechanism to ensure that what gets built is not just impressive in demonstration, but defensible in production. In practice, this means a few specific things:

• It means specifications that include non-functional requirements from the start, not as an afterthought once a prototype has already shaped the conversation. A PM who specifies latency thresholds, security requirements, audit logging standards, and error-handling behaviour before development begins is doing something qualitatively different from one who specifies features and leaves the rest to engineering.

• It means a clear, enforced distinction between prototype and production. With prompt-driven development, the phrase “it’s just a prototype” has a habit of becoming “we’re rolling it out next week” without any formal decision having been made. The technically grounded PM is the person who notices that transition happening and asks the governance question out loud: what review has this received, who has approved it, and what has changed between what was demonstrated and what is being deployed?

• It means an honest relationship with engineering’s invisible toil. The work of building production-safe software, which includes instrumentation, pipeline integration, security hardening, and resilience testing, is not to be minimised. A PM who understands that work, and can explain it to leadership in terms of risk rather than velocity, is protecting the long-term roadmap in a way that no prototype can do.

Therefore, as prompt-driven development continues to lower the barrier to building, where is the line between a high-fidelity prototype and production-ready software, and who in your organisation is responsible for that boundary? In most regulated institutions, the answer to the second part of that question is not yet clear enough. The technically grounded product manager should be.

Aiversight works with risk and compliance leaders in regulated financial services to identify where accountability gaps exist in technology delivery and to build governance frameworks that make fast-moving development defensible under scrutiny. If your organisation is navigating the tension between prototype speed and production safety, we are available to discuss it.