The Invisible Hand Cannot Hold the Guardrails
On the Structural Impossibility of AI Self-Regulation Under Capitalism
Abstract
This paper advances three linked claims. First, that meaningful self-regulation of artificial intelligence is structurally impossible within a capitalist economy, because the fiduciary, competitive, and short-horizon incentives that define publicly-traded and venture-backed firms make voluntary deceleration a self-eliminating strategy. Second, that the historical record of comparable industries — tobacco, leaded gasoline, chlorofluorocarbons, asbestos, pharmaceuticals, aviation, finance, and social media — provides a consistent, empirically verifiable pattern: every self-regulatory regime collapsed, and every industry denied known harms for decades while deploying them at scale. Third, that the dangerous frontier of any general-purpose technology is not explored first by its researchers or its commercial integrators, but by criminals and opportunists whose incentives are more elastic, whose ethical constraints are absent, and whose capital costs are lower — producing damage that vastly exceeds what thoughtful, proactive development would have caused.
The synthesis is pessimistic but bounded: reactive regulation always arrives, but it arrives after the body count. The policy question is not whether AI will be regulated, but how many harms we will normalize between now and then. We conclude with the conditions under which proactive governance has historically succeeded — and the narrow window in which those conditions still apply to AI.
Contents
- The Structural Case: Why Self-Regulation Cannot Work
- Historical Precedent: A Century of Failed Self-Regulation
- The Criminal Vanguard: Who Actually Leads Technological Arms Races
- Present Harm: AI Is Already Dangerous
- Short-Termism and the Architecture of Reactive Policy
- The Collingridge Dilemma and the Precautionary Alternative
- Conclusion: What the Evidence Demands
- References
Section I. The Structural Case: Why Self-Regulation Cannot Work
1.1 The fiduciary constraint
American corporate law, as consolidated in Dodge v. Ford Motor Co., 204 Mich. 459 (1919), establishes that the directors of a for-profit corporation owe their primary duty to shareholders, not to the public. The subsequent Delaware line of cases — Revlon, Inc. v. MacAndrews & Forbes Holdings, 506 A.2d 173 (Del. 1986), and Unocal Corp. v. Mesa Petroleum Co., 493 A.2d 946 (Del. 1985) — refined but did not soften this duty. Whatever the rhetorical commitments of an AI laboratory, its officers face a legally actionable obligation to maximize shareholder value, and voluntary unilateral slowdown in the face of profitable demand is, in the limit, a breach of that duty.
Public benefit corporations (PBCs) and "capped-profit" hybrid structures were widely invoked during the 2015–2023 period as a solution. OpenAI's original non-profit governance structure, Anthropic's PBC charter, and DeepMind's internal ethics board were each presented as evidence that mission could constrain market. The subsequent record is unambiguous: OpenAI's November 2023 board crisis ended with the removal of the directors who had invoked the mission clause; its 2024 restructuring moved the for-profit subsidiary toward conventional equity. The internal conclusion, reached under pressure, was that capital flows to the governance structure that optimizes for returns, and mission-first structures starve.
1.2 The collective-action problem
Even where individual actors sincerely prefer caution, the competitive structure of the industry forbids it. This is the classic n-player prisoner's dilemma: if any single lab pauses to validate, and the others do not, the pausing lab loses talent, capital, and strategic position. A safety-motivated pause is therefore individually irrational whenever it is collectively necessary. Nash equilibrium in a race without binding coordination is the race.
The "Pause Giant AI Experiments" open letter (Future of Life Institute, 22 March 2023), signed by over thirty thousand people including researchers at the signatories' competitors, asked for a six-month moratorium on training runs above GPT-4. No lab paused. Sam Altman, Dario Amodei, Demis Hassabis, and Elon Musk (a signatory who subsequently founded xAI to compete directly) continued scaling. The letter is not evidence that caution was absent from the field; it is evidence that caution cannot bind the field.
1.3 The capture of safety
When a firm's safety function conflicts with its revenue function, the former loses. OpenAI's Superalignment team, launched in July 2023 with a pledged 20% of compute, was effectively dissolved in May 2024 following the resignations of co-leads Ilya Sutskever and Jan Leike. Leike's public departure statement noted: "safety culture and processes have taken a backseat to shiny products" (X/Twitter, 17 May 2024). Google's Ethical AI team was decapitated earlier, with the forced departures of Timnit Gebru (December 2020) and Margaret Mitchell (February 2021) following the Stochastic Parrots paper. Microsoft dissolved its internal AI ethics and society team in March 2023 amid the Bing/Copilot push. Meta disbanded its Responsible AI team in November 2023.
The pattern is not coincidence. The function of an internal ethics team is to slow or block deployments; the function of a commercial product team is to ship. When the two conflict, the ship side wins, because shipping is what the firm is for. This is not a failure of individual courage; it is a failure of organizational design under competitive pressure.
1.4 The short-horizon constraint
Public equity markets discount cash flows beyond roughly five years sharply; venture capital typically underwrites on a seven-to-ten-year fund lifetime with liquidity expected earlier. Neither horizon accommodates the externalities of a technology whose societal effects — on labor markets, elections, misinformation, concentration of power — unfold over decades. A firm that internalizes those externalities at its own cost transfers capital to firms that do not. The price signal therefore systematically under-weights long-horizon harm, a failure formally characterized by the literature on time-inconsistent preferences (Laibson 1997) and the social cost of carbon (Nordhaus 2017).
Section II. Historical Precedent: A Century of Failed Self-Regulation
The argument that industries can police themselves is not an untested hypothesis; it has been tried, in public, in documented form, across every major technology of the twentieth century. The test has failed every time. What follows is a compressed record.
2.1 Tobacco (1953–1998)
On 4 January 1954, the major American tobacco companies published "A Frank Statement to Cigarette Smokers" in more than four hundred newspapers. It promised: "We accept an interest in people's health as a basic responsibility, paramount to every other consideration in our business." They established the Tobacco Industry Research Committee, ostensibly to fund independent science.
Documents released under the 1998 Master Settlement Agreement — the "Legacy Tobacco Documents Library," now hosted by UCSF — establish that by 1953 the industry's internal scientists had already confirmed the carcinogenicity of cigarette smoke. A 1953 memo by Claude Teague of R.J. Reynolds identified the tumorigenic fractions. The research committee's public function was denial; its internal function was intelligence-gathering about what the public might learn next. Forty-four years passed between internal confirmation and public settlement. During those forty-four years, the WHO estimates cigarettes killed on the order of one hundred million people worldwide.
2.2 Leaded gasoline (1923–1996, then 2021 globally)
Tetraethyl lead was added to gasoline by General Motors, DuPont, and Standard Oil's joint Ethyl Corporation beginning in 1923. Workers at the Standard Oil Bayway refinery in New Jersey began dying of acute lead poisoning within months; a 1924 outbreak killed five and hospitalized thirty-five, several driven psychotic. Industry chemist Thomas Midgley Jr. — later the inventor of CFCs — held a public press conference in which he washed his hands in tetraethyl lead to demonstrate its safety. He had already taken extended medical leave for lead poisoning.
The U.S. Public Health Service convened a 1925 conference; the industry's expert witnesses dominated, alternatives were dismissed, and lead was permitted in gasoline. Clair Patterson's 1965 geochemical work — originally aimed at dating the Earth — established background-to-modern lead ratios that proved contamination of the entire biosphere. Herbert Needleman's 1979 studies of tooth lead and childhood IQ demonstrated the neurotoxicity at population scale. Leaded gasoline was phased out in the U.S. over 1976–1996 and was not eliminated globally until Algeria ceased use in 2021.
2.3 Chlorofluorocarbons (1974–1987)
Mario Molina and F. Sherwood Rowland published "Stratospheric sink for chlorofluoromethanes" in Nature on 28 June 1974, predicting that CFC-11 and CFC-12 would degrade the ozone layer. DuPont, the dominant CFC manufacturer, responded that the theory was "science fiction." Joseph Farman's British Antarctic Survey paper (Nature, 16 May 1985) documented the ozone hole directly. The Montreal Protocol was signed 16 September 1987 — and succeeded only because substitutes (HFCs) were commercially viable for DuPont, aligning abatement with its competitive interest.
2.4 Asbestos (1930s–ongoing)
Internal correspondence at Johns-Manville documents awareness of asbestosis by the early 1930s. A 1934 letter from Vandiver Brown of Johns-Manville suppressed publication of asbestosis research in Asbestos magazine. Irving Selikoff's 1964 epidemiological studies established mesothelioma risk publicly. Johns-Manville declared bankruptcy in 1982. Asbestos remains legal in the United States in certain forms; the EPA's 2024 partial ban covers only chrysotile. Annual global asbestos deaths still exceed 200,000 according to WHO estimates. Interval from internal knowledge to even partial U.S. regulation: ninety years.
2.5 Pharmaceutical opioids (1996–present)
Purdue Pharma introduced OxyContin in 1996 with marketing claims that its time-release formulation made addiction "less than one percent" likely. Internal documents released in state litigation, and in the Justice Department's 2020 plea agreement, established that Purdue knew of widespread crushing-and-snorting abuse by 1997. The Sackler family extracted over ten billion dollars from Purdue before its 2019 bankruptcy. CDC data places U.S. opioid overdose deaths at over 800,000 between 1999 and 2023.
2.6 Financial services and the 2008 crisis
The credit rating agencies — Moody's, S&P, Fitch — functioned as self-regulators of structured finance. They were paid by the issuers of the securities they rated. The Financial Crisis Inquiry Commission (2011) documented that by 2006 internal analysts at Moody's had raised alarms; their managers overrode them. S&P downgraded $1.9 trillion of securities over 2007–2008, securities it had itself rated AAA. No senior executive at any rating agency, and no senior executive at any major bank, was criminally prosecuted.
2.7 Boeing 737 MAX (2011–2019)
Under the FAA's Organization Designation Authorization program, Boeing was permitted to self-certify elements of aircraft airworthiness. MCAS's reliance on a single angle-of-attack sensor was not flagged as safety-critical. Lion Air Flight 610 (29 October 2018, 189 dead) and Ethiopian Airlines Flight 302 (10 March 2019, 157 dead) followed. The House Committee on Transportation and Infrastructure's September 2020 report concluded that Boeing and the FAA both knew of design flaws before the second crash and failed to ground the fleet.
2.8 Social media (2004–present)
Frances Haugen's September 2021 disclosure — published as "The Facebook Files" by the Wall Street Journal — established that Meta's internal researchers had quantified Instagram's harm to teenage mental health, the amplification of ethnic violence in Ethiopia and Myanmar, and the measurable preference of the engagement algorithm for outrage content. Meta's public response denied, then minimized, then declined to act on, each finding. No substantive U.S. federal legislation has passed. Section 230 of the Communications Decency Act remains intact.
2.9 The pattern
Eight industries. Eight self-regulatory regimes. Eight internal-knowledge / public-denial periods ranging from thirteen years (CFCs, atypically short) to ninety years (asbestos). The shared features:
- Early internal confirmation of harm, often by the firm's own scientists.
- Public denial, funded counter-research, and captured advisory bodies.
- Regulatory action only after undeniable mass harm, typically forced by litigation or disaster rather than by policy foresight.
- Minimal individual accountability for the decision-makers involved.
AI exhibits all four features in their early stages. Internal safety teams are being disbanded. Public commitments to responsible scaling coexist with compute expansions of two orders of magnitude. Regulatory bodies are staffed by industry alumni. Accountability mechanisms are nominal. The null hypothesis — that AI will follow the same path — is supported by every comparable data point on record.
Section III. The Criminal Vanguard: Who Actually Leads Technological Arms Races
There is a persistent fiction in the policy literature that the frontier of any technology is defined by its reputable builders — the companies, the academics, the conference circuit. This fiction is empirically false and has been for at least a century. The frontier is defined by actors whose incentives are unconstrained by reputation, liability, or customer trust: criminals, state offensive-intelligence services, and opportunists. They arrive first because their adoption function is steeper: they face none of the costs that civilians internalize.
3.1 The pattern across earlier technologies
Radio. The illicit use of short-wave radio by rum-runners during Prohibition drove U.S. Coast Guard investment in radio direction-finding through the 1920s, predating most commercial applications of the same techniques (Office of Naval Intelligence records, 1925–1933). Pirate broadcasters in the North Sea (Radio Caroline, 1964) forced the first modern European broadcasting regulations.
The automobile. Prohibition-era bootleggers — whose modified Ford V8s ("whiskey cars") could outrun law enforcement — drove the initial civilian engine-tuning industry, and, through the 1947 founding of NASCAR, much of the American performance-vehicle culture. The criminal use of the car arrived before its suburban use.
Cryptography. Phil Zimmermann's Pretty Good Privacy (1991) was investigated by the U.S. government for export-control violations; its civilian adoption lagged its use by dissidents, journalists, and criminals. The first widespread non-state deployment of strong encryption to protect financial transactions was by drug traffickers and arms dealers.
The early internet. Unsolicited commercial email — spam — predates the commercial web. The Canter & Siegel "Green Card Lottery" Usenet spam of 12 April 1994 established the modern mass-unsolicited-messaging technique; the Morris worm (2 November 1988) established the first internet-scale self-replicating exploit. By the late 1990s, phishing, botnets, and credit-card fraud operations had matured faster than most corporate e-commerce security. The ILOVEYOU worm (2000) caused roughly $10 billion in damages globally. Defensive capability arrived after, not before.
Deepfakes. The term "deepfake" originates from a Reddit user of that handle who, in late 2017, used generative adversarial networks to swap celebrity faces onto pornographic video. Non-consensual sexual imagery was the first mass-deployment use case. The DeepNude app (June 2019) commercialized the same technique for image-based sexual abuse before any reputable firm productized face-swap tools. Detection research and platform policy followed.
Ransomware. The CryptoLocker campaign (September 2013) industrialized the asymmetric-crypto ransomware model. By 2017, EternalBlue — leaked from NSA's Tailored Access Operations in April 2017 — powered WannaCry (May 2017, ~200,000 systems across 150 countries) and NotPetya (June 2017, with damages exceeding $10 billion concentrated at Maersk, Merck, and Saint-Gobain). The worst cyberweapon deployments of the decade were criminal repurposings of state-built capability. Defenders were years behind.
3.2 The AI case so far
Criminal actors have moved faster on generative AI than most regulated industries. The documented record, through April 2026:
- WormGPT (identified by SlashNext, July 2023), a fine-tune of the open-source GPT-J model stripped of refusal behavior and marketed explicitly to phishing and business-email-compromise operators.
- FraudGPT (identified by Netenrich, July 2023), similarly marketed on dark-web forums with a subscription model and explicit documentation for carding, malware writing, and phishing-site generation.
- The Arup deepfake. In late January 2024, an employee of the British engineering firm Arup in Hong Kong transferred HK$200 million (approximately US$25.6 million) to attackers across fifteen separate transactions, following a video conference in which every other participant — including the firm's UK-based CFO — was a real-time deepfake. Hong Kong Police disclosed the incident in a 4 February 2024 briefing; Arup confirmed it as its own in May 2024. It was, at that date, the largest known single deepfake-enabled fraud.
- Voice cloning fraud. The FBI's Internet Crime Complaint Center (IC3) documented over US$2.9 billion in reported BEC/EAC losses for 2023, an increasing fraction involving AI-synthesized voice. Regional incidents — including the January 2020 UAE bank heist ($35 million via cloned director voice) and numerous documented family-emergency scams targeting elderly Americans — predate and continue through the current generation of commercial voice-cloning tools.
- AI-generated CSAM. The Internet Watch Foundation's October 2023 report "How AI is being abused to create child sexual abuse imagery" identified, in a single month of monitoring a single dark-web forum, 20,254 AI-generated CSAM images, 2,562 of which were assessed as criminal under UK law. Its July 2024 follow-up documented increasing realism and the first AI-generated videos. Open-weights image models released by well-funded labs were the primary generative substrate.
- Political deepfakes. The New Hampshire primary robocall of 21 January 2024 used a Joe Biden voice clone to suppress Democratic turnout; the operator was later identified and fined $6 million by the FCC. Comparable incidents have been documented in Slovakia (September 2023 election), Bangladesh (January 2024), and Pakistan (February 2024).
- Autonomous offensive tooling. Academic red-team demonstrations — including the "PentestGPT" line of work and multiple prompt-injection-enabled agent hijacks documented in 2024–2025 — show that the agentic capabilities commercial AI firms are racing to ship carry latent offensive utility immediately upon release. Criminal adoption is measured in weeks.
3.3 Why the vanguard is always criminal
The structural reason is simple and has been stable across technologies. Legitimate actors face four costs that criminal actors do not: regulatory compliance, brand risk, civil liability, and the opportunity cost of their existing revenue base. A bank cannot deploy a hastily-validated model in customer-facing fraud detection because an error is a lawsuit. A fraud ring faces no such cost; an error is a failed attempt. The result is that criminal operators iterate at a rate set by their own feedback loops, while legitimate operators iterate at the rate set by their slowest compliance review.
This asymmetry is compounded in AI by two features of the current ecosystem. First, model weights leak — by release (Llama series), by theft (internal Meta and other leaks), or by replication (open-source reproductions of closed models within 6–18 months). Second, the marginal cost of offensive application is near zero: a single GPU and a jailbreak prompt produce thousands of phishing variants. The offense-defense balance is therefore skewed far more toward offense than in any prior information-technology wave, and will remain so as long as frontier models are deployed at scale.
3.4 The implication
The common policy framing — "we must move fast so the good actors stay ahead of the bad" — inverts the empirical record. The bad actors are already ahead, because their cost structure allows them to be. Racing the frontier forward does not close the gap; it extends the attack surface. The only interventions that have historically closed comparable gaps involved slowing the rate at which offensive primitives became widely available: cryptographic export controls (partially effective 1991–2000), controlled-substance precursor regulation (partially effective), dual-use biological material controls. Each involved accepting real costs to legitimate users in order to impose larger costs on illegitimate ones. AI policy has so far refused this trade-off.
Section IV. Present Harm: AI Is Already Dangerous
The public debate remains oriented toward hypothetical future risks: AGI, superintelligence, existential catastrophe. This framing serves the firms most responsible for current systems, because it converts their present conduct into a preparatory stage for a future they claim to be uniquely qualified to manage. The framing is false. The documented harms of currently-deployed AI are substantial, measurable, and repeating.
4.1 Medical
Epic Systems' sepsis prediction model, deployed across hundreds of U.S. hospitals, was externally evaluated by Wong et al. (JAMA Internal Medicine, June 2021) at 27,697 patients across the University of Michigan system. The model missed 67% of sepsis cases and generated false alarms at a rate that produced clinician desensitization. The model had not been externally validated before widespread deployment.
The National Eating Disorders Association's "Tessa" chatbot, deployed to replace a human helpline in late May 2023 after the helpline staff unionized, was taken offline on 30 May 2023 after providing weight-loss and calorie-restriction advice to users seeking eating-disorder support. The replacement of the human service by an AI service preceded its safety validation.
4.2 Criminal justice
ProPublica's May 2016 analysis of the COMPAS recidivism model (Angwin, Larson, Mattu, Kirchner) documented that Black defendants were almost twice as likely to be misclassified as higher-risk than white defendants, and white defendants were more likely to be misclassified as lower-risk. COMPAS remains in use across numerous U.S. jurisdictions. The State v. Loomis decision (Wisconsin Supreme Court, 2016) permitted its use in sentencing, rejecting the defendant's due-process challenge.
Facial-recognition misidentification has produced documented wrongful arrests. Robert Williams (January 2020, Detroit), Nijeer Parks (February 2019, Woodbridge NJ), Randal Reid (November 2022, Louisiana via Georgia), and Porcha Woodruff (February 2023, Detroit; arrested while eight months pregnant) were each arrested on the basis of false facial-recognition matches. All four are Black; facial-recognition error rates for darker-skinned women have been documented at up to 34% in commercial systems (Buolamwini & Gebru, "Gender Shades," 2018).
4.3 Employment
Amazon's experimental hiring model, built beginning in 2014, was found by 2015 to be downranking applications containing the word "women's" (as in "women's chess club captain") and similar gender-correlated signals; the team was eventually dissolved in early 2017, and the episode was disclosed publicly via Reuters in October 2018. The model was never used for live hiring decisions, but the pattern — training on biased historical data reproduces and amplifies the bias — has reappeared in every subsequent audit of employment AI. HireVue, Pymetrics, and comparable vendors have faced state-level regulatory action in Illinois, New York, and Maryland.
4.4 Financial fraud and elder abuse
The FTC's Consumer Sentinel reported $10 billion in total consumer fraud losses for 2023, a record, with an increasing fraction involving AI-synthesized voice or text. AARP-aligned surveys place U.S. elder financial exploitation losses above $28 billion annually. Voice-cloning-enabled "grandparent scams" have become common enough that the FBI issued a public service announcement on 12 December 2024.
4.5 Misinformation at scale
Generative AI has reduced the marginal cost of producing plausible text, image, and video to near zero. NewsGuard's running count of "unreliable AI-generated news and information sites" (UAINS) exceeded 1,000 in 2024. The Stanford Internet Observatory and the Election Integrity Partnership documented persistent networks using LLM-generated content to seed propaganda during the 2024 U.S. election cycle. Platform-level detection remains unreliable.
4.6 Autonomous weapons and targeting
Israeli journalism outlets +972 Magazine and Local Call, in an April 2024 investigation, documented the use of a target-generation system referred to as "Lavender" by the IDF during the Gaza campaign. The reporting, based on interviews with six IDF intelligence officers, described a system that generated approximately 37,000 human targets with minimal per-target human review and an acknowledged error rate of roughly 10%. The Israeli military has disputed aspects of the reporting but confirmed the use of AI target-generation. Whatever one's view of the underlying conflict, the use of statistical target-generation systems to authorize lethal force, with minimal per-case review, is a present-tense fact.
4.7 Concentration of economic power
Training frontier models now requires capital expenditure in the billions of dollars. Microsoft committed $80 billion in AI-related capex for FY2025; Google, Meta, and Amazon each committed comparable sums. The OpenAI/Oracle/SoftBank "Stargate" announcement of January 2025 proposed $500 billion in U.S. AI infrastructure over four years. The capital intensity of the frontier has foreclosed entry to all but a handful of firms, and has tied those firms to hyperscale cloud providers as both customers and competitors. Whatever the technology's eventual utility, the market structure it is producing is oligopolistic, and the political economy implications — of having four or five firms control the substrate of future economic activity — have not been meaningfully debated.
4.8 The accounting
These harms are not speculative. They are measured in wrongful arrests, in cardiac deaths from missed diagnoses, in $25 million wire transfers, in 200,000 propaganda sites, in 20,000 CSAM images per month from a single forum, in 37,000 algorithmic targets. The rhetorical move that treats all of this as preamble to a future discussion of "real" AI risk is itself a form of harm: it launders present damage by reframing it as the cost of progress toward a promised future that the historical record (Section II of the companion paper, The AI Disaster) suggests will not arrive.
Section V. Short-Termism and the Architecture of Reactive Policy
5.1 The quarterly discount
Capital markets are not neutral accounting instruments. They are machines that assign differential weight to near-term and long-term cash flows, and they do so according to investor preferences that have been documented as shorter than socially optimal across every major study of the question. Jackson (2005), the Kay Review (2012, commissioned by the UK Department for Business), and the CFA Institute's recurring "Investor Trust" surveys converge on a time horizon for activist public-equity investors of eighteen to thirty-six months. No technology whose principal externalities unfold over twenty years can be rationally self-regulated by firms operating on that horizon.
5.2 The venture function
Venture capital has a different but equally short constraint. A typical 10-year fund is expected to return capital within 7–10 years, with the best returns coming from the roughly 10% of investments that become outsized exits. This shapes portfolio-company behavior: the incentive is to grow explosively toward a liquidity event, not to mature cautiously toward a durable business. Anthropic's $61.5 billion valuation (March 2025), OpenAI's $300 billion valuation (March 2025), xAI's $75 billion valuation (May 2025), Safe Superintelligence's $32 billion valuation with no product (September 2025), and Mira Murati's Thinking Machines Lab at $12 billion pre-product (July 2025) are not aberrations. They are the VC asset class functioning correctly on its own terms. Those terms are not public interest.
5.3 Lobbying spend and regulatory capture
OpenSecrets data shows U.S. lobbying on AI issues rose from a handful of disclosures in 2017 to several hundred organizations disclosing AI-related lobbying by 2024, with collective spend in the hundreds of millions of dollars. The Biden White House's Executive Order 14110 (30 October 2023) was negotiated heavily with industry and imposed only reporting requirements on the largest training runs; it was rescinded by President Trump's Executive Order 14179 (23 January 2025), which explicitly removed reporting obligations. California SB 1047, which would have imposed liability on developers of frontier models, was vetoed by Governor Newsom on 29 September 2024 after sustained industry opposition. The EU AI Act (Regulation 2024/1689), in force from 1 August 2024, is the most substantive global regime; its risk-tiered structure was heavily shaped by industry input and its enforcement capacity is not yet demonstrated.
5.4 The pacing problem
The "pacing problem" — the observation that law always lags technology — is not a neutral fact. It is a policy choice. Adam Thierer's formulation of the problem (2018) treats it as an argument for permissive regulation, on the premise that rulemaking cannot keep up with innovation. A different reading is available: the pacing gap exists because regulation is reactive by design, triggered by documented harm, and because the affected publics lack the technical literacy to demand otherwise. Pre-market review — required for drugs, aircraft, and medical devices — closes the pacing gap by placing the burden of proof on the deployer. The political choice not to apply pre-market review to AI is not the result of pacing; pacing is the result of the choice.
5.5 Reactive policy's body count
Reactive regulation is not cheap. The deaths required to produce the FDA's 1962 Kefauver-Harris Amendments (Thalidomide: over 10,000 children born with severe birth defects globally); the deaths required to produce NHTSA (Ralph Nader's Unsafe at Any Speed, 1965, followed tens of thousands of annual U.S. auto fatalities); the deaths required to produce meaningful workplace chemical regulation (Rachel Carson's Silent Spring, 1962, followed decades of documented poisoning); the deaths required to ground the 737 MAX (346). Every regulatory regime we now take as normal was paid for in advance, in bodies.
The question is not whether AI will produce its version of this debt. It is whether we are willing, this time, to pay the debt forward — to accept the costs of proactive caution before the documented casualty count reaches the threshold that historically has been required to force action. The structural argument of Section I suggests we will not, absent external intervention.
Section VI. The Collingridge Dilemma and the Precautionary Alternative
6.1 The dilemma
David Collingridge, in The Social Control of Technology (1980), formulated what is now called the Collingridge Dilemma: in the early stages of a technology's development, its eventual social impact cannot be reliably predicted and therefore cannot be rationally regulated; by the time its social impact is visible, the technology has become entrenched and regulation has become prohibitively expensive. The dilemma is usually cited as a counsel of humility. It is more useful read as a description of the narrow window in which intervention is both knowable and tractable, and an argument for expanding that window deliberately.
6.2 The precautionary principle, correctly specified
The precautionary principle has been caricatured — often by parties with commercial interests in its dismissal — as a blanket rule against novel technology. Its actual formulation, as in the 1992 Rio Declaration (Principle 15) and the EU's consolidated application, is narrower: where an activity raises threats of serious or irreversible harm, lack of full scientific certainty shall not be used as a reason for postponing cost-effective measures to prevent it. The burden of proof shifts toward the deployer. This is not a novel regulatory posture; it is the default for pharmaceuticals, civil aviation, nuclear power, and genetically-modified organisms. The anomaly is that AI has been treated as its own exception.
6.3 What proactive regulation looks like when it has worked
Four historical cases illustrate the conditions under which proactive regulation has succeeded:
- Asilomar (1975). Molecular biologists, confronted with the early results of recombinant-DNA work, convened at Asilomar in February 1975 under the chairmanship of Paul Berg and instituted a voluntary moratorium on classes of experiments, followed by NIH guidelines. The moratorium held because the community was small, the concern was concrete, and the funding sources (NIH) aligned with the caution.
- The Montreal Protocol (1987). Succeeded because HFC substitutes aligned DuPont's commercial interest with abatement.
- Nuclear non-proliferation. Imperfect but durable, maintained by a combination of supply-side controls (NSG, Wassenaar), demand-side incentives (NPT safeguards), and near-taboo normative work.
- Human germline genetic modification. After He Jiankui's 2018 CRISPR-edited births, the international scientific community's response was broad condemnation, prosecution under Chinese law, and the subsequent WHO expert advisory committee's 2021 framework. The taboo has, for now, held.
Three features recur across the successful cases: (1) early identification of specific harms before mass deployment; (2) alignment of commercial incentives with caution, either by subsidy, by substitute availability, or by liability exposure; (3) a normative community willing to treat violations as disqualifying. AI currently has none of the three in adequate measure.
6.4 Concrete instruments
A non-exhaustive list of instruments that have worked in comparable settings:
- Pre-market review for any AI system deployed in healthcare, criminal justice, employment, housing, credit, insurance, or public benefits — analogous to FDA premarket approval.
- Strict liability for harms caused by frontier-model deployments, with no safe harbor for "reasonable safety efforts" above a defined capability threshold.
- Compute reporting and licensing for training runs above a documented threshold (cf. EO 14110's withdrawn 1026-FLOP reporting).
- Mandatory external audits by accredited bodies, with findings disclosed publicly on a statutory schedule.
- Whistleblower protection specific to AI safety research, analogous to Sarbanes-Oxley Section 806.
- Prohibition of specific use cases: social scoring by governments, emotion-recognition in employment and education, untargeted scraping for biometric databases — each of which the EU AI Act prohibits and the U.S. does not.
- Data-provenance requirements for training corpora, with enforceable rights of removal and compensation for included creators.
- A statutory duty of care for large-language-model providers, as proposed in the UK Online Safety Act's original drafting.
Section VII. Conclusion: What the Evidence Demands
The evidence assembled here is not novel. Each element — the fiduciary constraint, the collective-action problem, the eight historical cases, the criminal-vanguard pattern, the present-tense harm catalog, the short-horizon incentive — is documented in primary sources that have been available for years. What is novel is the speed at which these factors are compounding in a single technology.
The central claim can now be stated plainly. It is not that the firms developing AI are unusually malevolent. Most of the individuals in them are not. It is that no collection of well-intentioned individuals operating under the incentive structure described in Section I can, in aggregate, produce cautious development. The structure selects against them. The ones who slow down lose, and the ones who do not lose shape the field.
The historical record described in Section II shows that this dynamic has played out, in recognizable form, across every major industrial technology of the twentieth century. In every case, internal knowledge of harm preceded public acknowledgement by decades. In every case, regulation arrived after mass casualties had made continued denial politically untenable. In every case, the decision-makers responsible faced minimal individual consequences. There is no plausible reading of that record in which AI is the exception.
The arms-race framing described in Section III shows that the "move fast so good actors stay ahead" argument inverts the documented structure of technological arms races. The frontier is always occupied first by actors whose cost structure tolerates experimentation that reputable firms cannot. Acceleration of the legitimate frontier accelerates the illegitimate frontier with it, and by a larger margin, because the offense-defense asymmetry favors offense in almost every domain where AI is being deployed.
The present-harm catalog in Section IV makes the question of "whether AI is dangerous" moot. It is. The question is whether current and foreseeable harms are being weighed honestly against claimed benefits, and the structural answer from Section V is that they cannot be, because the weighing is being done by institutions whose horizon is too short and whose incentives are misaligned.
The alternative described in Section VI is not untried. Every element — pre-market review, strict liability, compute reporting, external audit, whistleblower protection, prohibited-use lists, provenance requirements, duty of care — exists, in mature form, in some existing regulatory regime. The barrier to their adoption for AI is not technical. It is political, and it is the direct consequence of the lobbying spend and regulatory capture documented in Section 5.3.
The policy posture that follows from this evidence is uncomfortable for most of the actors who would have to implement it, and comforting for none of them. It is also the only posture supported by a century of comparable data. Capitalism produces remarkable artifacts. It has never, in any documented case, regulated them before they killed enough people to force the question. The argument of this paper is that we already have enough documentation of harm to force the question now, without waiting for the corresponding body count — and that history suggests we will wait anyway, unless external action changes the incentive structure the industry cannot change from within.
The window for that external action is the subject of a separate paper. It is narrower than the industry's own timelines suggest, and narrower than policymakers currently assume. The cost of missing it is the cost the historical record has already shown us: decades of denial, measurable harm compounding, and the eventual, too-late regulation that follows only the dead.
References and Primary Sources
Corporate-law foundations
- Dodge v. Ford Motor Co., 204 Mich. 459, 170 N.W. 668 (1919).
- Revlon, Inc. v. MacAndrews & Forbes Holdings, Inc., 506 A.2d 173 (Del. 1986).
- Unocal Corp. v. Mesa Petroleum Co., 493 A.2d 946 (Del. 1985).
- State v. Loomis, 881 N.W.2d 749 (Wis. 2016).
Tobacco
- Tobacco Industry Research Committee, "A Frank Statement to Cigarette Smokers," 4 January 1954.
- Legacy Tobacco Documents Library, University of California San Francisco, industrydocuments.ucsf.edu/tobacco.
- Master Settlement Agreement, 23 November 1998.
Leaded gasoline and CFCs
- Kovarik, W., "Ethyl-leaded gasoline: how a classic occupational disease became an international public health disaster," International Journal of Occupational and Environmental Health, 2005.
- Molina, M.J. & Rowland, F.S., "Stratospheric sink for chlorofluoromethanes: chlorine atom-catalysed destruction of ozone," Nature, 249, 810–812 (1974).
- Farman, J.C. et al., "Large losses of total ozone in Antarctica reveal seasonal ClOx/NOx interaction," Nature, 315, 207–210 (1985).
- Needleman, H.L. et al., "Deficits in psychologic and classroom performance of children with elevated dentine lead levels," New England Journal of Medicine, 300:689–695 (1979).
Asbestos, opioids, finance, aviation
- Selikoff, I.J. et al., "Asbestos exposure and neoplasia," JAMA, 188:22–26 (1964).
- U.S. Department of Justice, "Purdue Pharma L.P. Pleads Guilty to Fraud and Kickback Conspiracies," press release, 24 November 2020.
- Financial Crisis Inquiry Commission, Final Report, January 2011.
- House Committee on Transportation and Infrastructure, Final Committee Report on the Design, Development & Certification of the Boeing 737 MAX, September 2020.
Social media and AI safety institutions
- Wall Street Journal, "The Facebook Files," multi-part series, September–October 2021.
- Bender, E.M., Gebru, T., McMillan-Major, A. & Shmitchell, S., "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?," FAccT '21.
- Future of Life Institute, "Pause Giant AI Experiments: An Open Letter," 22 March 2023.
- Leike, J., departure statement, X/Twitter, 17 May 2024.
Criminal and fraudulent AI use
- SlashNext, "WormGPT – The Generative AI Tool Cybercriminals Are Using," July 2023.
- Netenrich, "FraudGPT: The Villain Avatar of ChatGPT," July 2023.
- Hong Kong Police Force, public briefing on Arup deepfake incident, 4 February 2024.
- Internet Watch Foundation, "How AI is being abused to create child sexual abuse imagery," October 2023; follow-up report, July 2024.
- FBI Internet Crime Complaint Center, Internet Crime Report 2023.
- FCC, "In the Matter of Lingo Telecom, LLC," enforcement action on New Hampshire deepfake robocall, 2024.
Present-harm documentation
- Angwin, J., Larson, J., Mattu, S., Kirchner, L., "Machine Bias," ProPublica, 23 May 2016.
- Buolamwini, J. & Gebru, T., "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification," FAT* (2018).
- Wong, A. et al., "External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients," JAMA Internal Medicine, 181(8):1065–1070 (2021).
- Dastin, J., "Amazon scraps secret AI recruiting tool that showed bias against women," Reuters, 10 October 2018.
- Abraham, Y., "‘Lavender’: The AI machine directing Israel's bombing spree in Gaza," +972 Magazine / Local Call, 3 April 2024.
Policy and regulation
- Executive Order 14110, "Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence," 30 October 2023 (rescinded).
- Executive Order 14179, "Removing Barriers to American Leadership in Artificial Intelligence," 23 January 2025.
- Regulation (EU) 2024/1689 (EU AI Act), in force 1 August 2024.
- California SB 1047, "Safe and Secure Innovation for Frontier Artificial Intelligence Models Act," vetoed 29 September 2024.
- Collingridge, D., The Social Control of Technology (Frances Pinter, 1980).
- United Nations Conference on Environment and Development, Rio Declaration, Principle 15, 1992.
- Berg, P. et al., "Summary Statement of the Asilomar Conference on Recombinant DNA Molecules," PNAS, 72(6):1981–1984 (1975).
This paper is part of the IXN.AI research series on the political economy of artificial intelligence. Companion paper: The AI Disaster: Why Artificial Intelligence Fails.