Personal Learnings← Reading Room

Tech & AI

AI Snake Oil

Arvind Narayanan & Sayash Kapoor

23 issues · 20 keepers · 12 tier-5 · 8 tier-4

AI Agents — Reliability & Benchmark Reform

2 tier-5 · 1 tier-4

Across this cluster Narayanan and Kapoor build the case that the way the field measures AI agents is broken, and that the broken measurement is *why* agents impress on leaderboards while disappointing in deployment. Their core moves: accuracy is the wrong sole metric (cost must be co-optimized via Pareto curves), benchmarks reward shortcuts and irreproducible setups, and "average task success rate" hides the consistency/robustness/calibration failures that actually block economic impact. Reliability — not raw capability — is the research bottleneck, and they propose concrete reforms and metrics to make it measurable.

AI leaderboards are no longer useful. It's time to switch to Pareto curves.

TIER 5 Apr 30, 2024

Accuracy-only leaderboards for AI agents are actively misleading: agents can invoke models thousands of times, so accuracy without cost is an unbounded optimization target that rewards spending, not engineering. The fix is Pareto curves plotting accuracy against dollar cost.

Kapoor, Stroebl, and Narayanan (Princeton) re-ran three top-ranked HumanEval agents — LDB, LATS, Reflexion — against simple baselines: retry (up to five attempts at temperature zero), warming (gradually increasing temperature across retries), and escalation (start cheap with Llama-3 8B, step up to GPT-4 only on failure). No complex agent outperforms the warming baseline in accuracy, yet LATS costs over 50x more and Reflexion/LDB cost 50%+ more. Escalation beats LDB on both accuracy and cost. No prior paper tested any of these baselines.

Parameter count, the standard proxy for cost, misleads downstream users — Mistral's chart for Mixtral 8x22B uses active parameters, which flatters MoE architectures relative to their actual API price. Model evaluation (comparing training choices) and downstream evaluation (procurement) are distinct: only the latter needs dollar cost, and it needs exactly that.

Reproducibility failures make things worse. LATS evaluated on a HumanEval subset while claiming "all 164." LDB's code was generated by GPT-4 though the paper states GPT-3.5. Reproduced accuracies fell 3–4 points below reported figures. Andrew Ng's claim that "GPT-3.5 in an agent loop reaches 95.1%" follows from this misattribution. The proposed standard: Pareto curves with token counts published so readers can reprice, plus HELM-style standardization for agent evaluations.

AI agentsPareto curvescost-controlled evaluationleaderboardsreproducibility

New paper: AI agents that matter

TIER 5 Jul 3, 2024

AI agent benchmarks reward benchmark performance over real-world utility — simple repeated sampling beats complex agent architectures on HumanEval at lower cost. Reliability compounds fatally: at 2% per-call failure rate, a multi-step agent becomes useless. The paper proposes five fixes: control for inference cost, visualize accuracy-vs-cost as Pareto curves, distinguish model from downstream benchmarks, prevent benchmark shortcuts via proper hold-outs, and fix reproducibility errors that inflate accuracy estimates. Products like Rabbit R1, Humane AI pin, and Devin have all flopped, confirming the gap.

AI agentsagent benchmarksevaluation methodologyreliabilityreproducibility

New Paper: Towards a science of AI agent reliability

TIER 4 Feb 24, 2026

Nearly two years of rapid AI capability gains have produced only modest reliability improvements — and this gap explains why agents' economic impact remains gradual despite crushing benchmarks. Rabanser, Kapoor, and Narayanan (Princeton) decompose reliability into four dimensions borrowed from aviation and nuclear safety: consistency (same result on repeated runs), robustness (holds under rephrased instructions), calibration (knowing when you're wrong), and safety (bounded failures). Testing 14 models from OpenAI, Google, and Anthropic across 500 runs on GAIA and TauBench, they find consistency scores ranging 30–75%, calibration near chance on one benchmark, and no meaningful gap between providers. Larger models improve calibration but can hurt consistency. Automation tools (unattended workflows) need reliability thresholds before deployment; augmentation tools get a discount because humans review output. None of the four dimensions is solved.

AI agentsreliabilityevaluationbenchmarkscalibration

Capability Claims, Scaling & the Limits of AGI

2 tier-5 · 2 tier-4

This is the authors' direct assault on trend-extrapolation AGI forecasting. They dismantle the assumption that scaling laws plus a rising curve equal imminent superintelligence: scaling predicts perplexity, not the "emergent abilities" people care about; data is exhausting; AGI is not even a coherent milestone to scale *toward*; and the heuristics used to reason about capability (Moravec's paradox, scaling extrapolations, model-flip-flopping forecasts) are far weaker than they appear. The throughline: capability prediction is unreliable, and you don't need it anyway because diffusion is slow.

AI scaling myths

TIER 5 Jun 27, 2024

Scaling laws only measure perplexity — how well models predict the next token — not emergent capabilities, which follow no law-like pattern. LLMs may simply plateau once all representable tasks are in the training data. Data constraints are already biting: YouTube's 150 billion minutes yields far less than Llama 3's 15 trillion training tokens once music, silent video, and quality filtering are applied. Synthetic data doesn't solve this; it fixes specific gaps (math, code, alignment) but can't replace human pre-training data at volume — unlike AlphaGo's self-play, which only works in closed, verifiable environments. Meanwhile, market pressure is pushing models smaller: GPT-4o costs 75% less than GPT-4; Claude 3.5 Sonnet is 5x cheaper than Opus and more capable. Training compute keeps rising because smaller models require more FLOPs to reach equivalent performance. Scaling alone will not produce AGI.

scaling lawsemergenceAGItraining datasynthetic data

AGI is not a milestone

TIER 5 May 1, 2025

Declaring AGI has been achieved is not an actionable event — it carries no implications for businesses, policymakers, or safety, because AGI is not a milestone in any meaningful sense.

The nuclear weapons analogy fails on both counts: building a bomb was immediately observable and had immediate geopolitical impact. AGI has neither property. Whether a system constitutes AGI can only be determined retrospectively, because the link between a system's capabilities and its real-world impact depends entirely on diffusion, which happens at human timescales — decades, not months. Electricity, computing, and the internet each required decades-long diffusion; the Industrial Revolution averaged below 3% annual growth because adoption, not invention, was the bottleneck.

The proliferation of AGI definitions (Jasmine Sun catalogs over 20) is itself diagnostic. Impact-based definitions (e.g., OpenAI's 2018 "outperforms humans at most economically valuable work") describe a state of the world, not a property of an AI system, so they cannot be tested on a system in isolation. Internals-based definitions are unobservable. Benchmark-based definitions produce hill-climbing toward test performance that doesn't transfer to real-world utility.

Conflating capability with power corrupts safety reasoning. Dario Amodei's description of "powerful AI" slides from capability properties (proving theorems, writing code) into power properties (ordering materials, directing experiments) — but power is a design choice humans make about the environment AI operates in, not an intrinsic feature of the system.

On geopolitics, Jeffrey Ding's work on general-purpose technologies shows that diffusion capacity, not first-mover invention, determines long-run competitive advantage. China lags the US not in model capabilities (6–12 months behind) but in digitization, cloud adoption, and workforce training — the actual diffusion preconditions. Export controls that delay China by months are inconsequential in a decades-long diffusion game.

AGIcapability vs powerdiffusionsuperintelligenceAI milestones

Is AI progress slowing down?

TIER 4 Dec 18, 2024

Declaring model scaling dead is premature — the sudden narrative reversal from OpenAI, Anthropic, and Google in late 2024 reflects vested interests more than new evidence. "Scaling as usual" ended with GPT-4 class models because readily available data is exhausted, but that was already known; it doesn't prove no path forward exists. Ilya Sutskever, who talked up scaling when OpenAI needed fundraising, now emphasizes data limits from his startup Safe Superintelligence — a pattern illustrating why industry insiders shouldn't be trusted as forecasters. They hold at most a few months' proprietary advantage over outsiders, and their predictions have a poor track record (see: a decade of self-driving overoptimism). Their technical expertise also produces forecasts that underweight business and social factors.

Inference scaling — spending more compute at runtime, as in OpenAI's o1 and DeepSeek R1 — is real but domain-constrained. It helps where answers are verifiable and symbolic reasoning applies: math, coding, physics, LSATs. It doesn't help with writing or low-resource translation. The authors' paper "Inference Scaling fLaws" finds that repeatedly sampling until a verifier approves degrades performance after roughly 10 attempts when the verifier is even slightly imperfect. Reasoning models also underperform in agentic settings — on CORE-Bench, Claude 3.5 Sonnet scores 38% versus o1-mini's 24%. Unlike model scaling, inference scaling requires algorithmic advances, not just hardware, making progress less predictable.

The capability debate is largely beside the point. Product development already lags behind existing capabilities — the web's infrastructure was set in the mid-1990s but transformative products took another decade — and adoption lags products further still. The bottleneck for real-world AI impact is the capability-reliability gap and organizational adoption, not benchmark scores.

AI scalinginference scalingAI progressforecastingcapability vs impact

Fact checking Moravec's paradox

TIER 4 Jan 29, 2026

Moravec's paradox — that tasks hard for humans are easy for AI and vice versa — has never been empirically tested. The pattern is a selection effect: AI researchers focus on two interesting quadrants of a 2x2 matrix while ignoring tasks trivially easy or provably intractable for both. Restricting to interesting tasks guarantees the negative correlation appears.

The evolutionary story (sensorimotor skill = billion years of experience; reasoning = thin new veneer) doesn't hold either. Open-ended reasoning requires common sense, which is itself supposedly hard for AI. STRIPS-era symbolic systems and IBM's Watson both collapsed outside narrow, closed domains.

The paradox generates false alarm (imminent superhuman reasoning, white-collar bloodbath) and false comfort (robotics won't break through fast). Computer vision was once the canonical "hard for AI" case — until deep learning flipped it in 2012. New capabilities diffuse slowly — self-driving cars were tested for 15 years before policy reacted — so adaptation beats prediction.

Moravec's paradoxcapability predictionAI hyperoboticsdiffusion

AI in Science & the Reproducibility Crisis

2 tier-5 · 0 tier-4

The authors' home turf as researchers: the use of machine learning *inside* science, and the methodological rot it can spread. One essay documents data leakage as a cross-disciplinary reproducibility crisis spanning hundreds of papers, fed by publish-or-perish incentives and hype that suppresses skepticism; the other inverts the usual optimism to argue AI may *slow* science by flooding a non-market system with papers while the real bottleneck (turning output into progress) goes untouched. Together they argue AI-for-science funding may be aimed at the wrong target.

Scientists should use AI as a tool, not an oracle

TIER 5 Jun 3, 2024

ML-based science has a pervasive reproducibility crisis driven by leakage — training data contaminating test sets. Narayanan and Kapoor's Princeton database covers ~650 affected papers across 30 disciplines; where researchers look systematically, the majority of sampled studies commit the error. Two 2023 Nature papers claiming 2.2 million AI-discovered materials collapsed on inspection: the 41 synthesized compounds were mostly misidentified or already known. The root cause is hype suppressing skepticism: researchers tweak failing models rather than accepting predictability limits, and don't check suspiciously good results for leakage. Hockey-stick AI adoption across all fields since 2015 signals funding-chasing, not methodology change, and will worsen the crisis before it improves.

reproducibility crisisdata leakageML-based scienceREFORMS checklistAI hype

Could AI slow science?

TIER 5 Jul 16, 2025

Scientific publications have grown 500-fold since 1900 and now double every 12 years, yet actual progress has stayed flat or declined — the production-progress paradox. Park et al. find disruptive papers constitute a shrinking share of total output; Bloom et al. show that exponential growth in researchers is offset by falling output per researcher; Matt Clancy finds the share of Nobel-winning discoveries published in the preceding 20 years fell from 90% in 1970 to 50% in 2015.

The most plausible explanation is that overproduction is self-defeating: when the annual flood of papers is too large to track, attention collapses onto already-famous work. Chu and Evans show empirically that high publication volume entrenches top-cited papers and buries novel ones. Publish-or-perish incentives reinforce the trap, pushing researchers toward safe incremental work.

AI is likely to worsen each dynamic. First, it accelerates paper output, deepening the attention problem. Second, science's software infrastructure is already decades behind industry — 93% of biomedical papers that pledged to share code did not, and AI has already seeded errors in over 600 studies across 30 fields. Third, AI-based modeling excels at prediction without explanation. Just as epicycles made the geocentric model accurate while blocking the shift to heliocentrism, AI could let fields achieve better predictions from wrong theories, delaying paradigm change. Fourth, mathematician William Thurston's account of foliations illustrates why human understanding is the real output of science: when that understanding is bypassed, fields hollow out even as publication counts climb.

The remedies are structural — funding metascience, tying career advancement to theory and synthesis rather than paper counts, and building AI tools that target error-detection and understanding rather than literature generation. Accelerating production without fixing these bottlenecks is adding highway lanes when the slowdown is a toll booth.

AI in sciencemetasciencereproducibilityproduction-progress paradoxscientific progress

From Capability to Impact — Products, Jobs & Commercial Return

2 tier-5 · 0 tier-4

The economic counterweight to the hype: even where capability is real, the chain from "model can do X" to "industry is transformed / workers replaced / money is made" is long and full of friction. One essay anatomizes why generative AI has produced little commercial return (the "big five" barriers and a missing product-market-fit discipline); the other shows why software engineers — the most AI-exposed, most AI-adopting workers — are not being replaced, using a "decide-execute-deliver" model and unmasking "AI washing" of ordinary layoffs.

AI companies are pivoting from creating gods to building products. Good.

TIER 5 Aug 19, 2024

AI companies misread the market after ChatGPT: OpenAI and Anthropic built models without products (ChatGPT's iOS app came six months late), while Google and Microsoft shoved AI into everything without fit, producing Sydney and Gemini image disasters. The OpenAI boardroom fight was fundamentally about shifting from AGI-building to products. Five barriers remain: cost (inference dropped 100× in 18 months, but cheaper models just enable more retries, trading cost for accuracy); reliability (90% accuracy fails consumer expectations of determinism); privacy (useful assistants need emails and documents, far more sensitive than chat); security (AI worms via prompt injection are theoretically possible and under-addressed); and UI (natural-language interfaces leave no room for error correction). Even if all five are solved, organizational integration runs on a decade-long timescale.

product-market fitLLM reliabilityAI productscost vs accuracyAI bubble

Why AI hasn’t replaced software engineers, and won’t

TIER 5 Jun 11, 2026

Mass layoffs attributed to AI in software are overwhelmingly corporate AI-washing. Block's Jack Dorsey framed 4,000 cuts as AI-driven; reporting revealed pandemic-era overhiring under financial pressure. Snap's CEO cited "65% AI-written code" alongside 1,000 layoffs following an activist investor campaign. New York's WARN Act filings found one of 160+ companies checked the AI box — two-tenths of a percent of laid-off workers. Among executives surveyed, a 10x gap: 21% made large anticipatory cuts, only 2% linked cuts to actual AI.

Software engineer employment is still growing — 3 points per year below the no-AI counterfactual, per Federal Reserve economists.

The structural explanation is the "decide-execute-deliver sandwich." Writing code was never the bottleneck — developer coding time runs 9–61% of total work. AI compresses the execution middle, but the two ends resist automation: deciding what to build requires interpreting user needs and organizational priorities; delivering requires accountability liability law enforces. A GitHub study across 100,000 developers found AI produced an 8x increase in code volume but only 30% more releases.

"Vibe coding" (unsupervised) and "agentic engineering" (human accountable) are routinely conflated. SWE-chat found 44% of agent-produced code survives into commits; vibe-coded commits introduce vulnerabilities at 9x the human rate.

Demand is likely to grow: software is price-elastic, so cheaper production expands total quantity built — programmer employment grew from near-zero in 1950 to millions today. Fred Brooks argued in *No Silver Bullet* (1986) that AI removes accidental complexity but cannot touch essential complexity — correctly specifying what software should do.

AI and jobssoftware engineeringAI washingdecide-execute-deliveragentic engineering

AI as Normal Technology — The Framework & Its Domains

1 tier-5 · 2 tier-4

The spine of the whole publication. The 15,000-word manifesto recasts AI as a controllable tool that diffuses through society on decades-long timescales — not a separate superintelligent species — and the long causal chain from capability to impact becomes the source of human leverage. A companion FAQ clears up the predictable misreadings ("normal" ≠ mundane). A third piece stress-tests the framework against a specific domain (law), showing exactly why capability gains stall against regulatory, adversarial, and human-in-the-loop bottlenecks. Read the manifesto first; the others depend on it.

AI as Normal Technology

TIER 5 Apr 15, 2025

AI will behave like past general-purpose technologies — transformative but slow, shaped by institutions, and controllable — not as a quasi-species racing toward superintelligence. Narayanan and Kapoor cast this as description, prediction, and prescription simultaneously.

The argument turns on three timescales: AI methods (fast), applications (slower), and adoption/diffusion (decades). Epic's sepsis tool, trained on a feature unavailable at deployment, illustrates the capability-reliability gap that slows consequential adoption regardless of benchmark scores. GPT-4 in the top 10% of bar exam takers tells us little about automating legal practice — the exam overweights retrieval, underweights judgment. Electrification took 40 years to reach productivity statistics; a 2024 study finds 40% U.S. adult generative-AI usage translates to only 0.5–3.5% of work hours.

"Superintelligence" conflates capability with power. Human power has always come from tools, not biology. Control over AI is far more intervenable than alignment discourse assumes — via auditing, circuit breakers, least-privilege design, and hierarchical control from cybersecurity. For geopolitical forecasting or persuasion against self-interest, trained human teams are predicted to approach an irreducible error floor AI cannot meaningfully beat.

Arms races are sector-specific: Waymo's safety culture beat Cruise and Uber, and the pattern yields to regulation. Model alignment cannot anchor misuse defense because harmful intent lives in orchestration code outside the model — defenses must sit downstream. Catastrophic misalignment is "speculative": genuine uncertainty exists about whether the risk is nonzero at all. The more probable systemic risks — inequality, labor displacement, democratic erosion — are normal byproducts of capitalism amplified by a powerful tool.

On policy, expected-utility calculations are ungrounded since AI risk probabilities vary by orders of magnitude with no valid reference class. The recommended posture is resilience — decentralization, open models, downstream defenses, preserved institutions — not nonproliferation, which creates monocultures and concentrates the power it claims to prevent.

AI as normal technologyAI policydiffusionexistential riskresilience

AI Won’t Automatically Make Legal Services Cheaper

TIER 4 Feb 12, 2026

Even if AI passes the bar, it will not automatically make legal services cheaper. Three structural bottlenecks block capability from reaching affordable client outcomes.

First, unauthorized practice of law (UPL) regulations — a felony in some jurisdictions — prohibit non-lawyers from applying legal knowledge to specific circumstances. AI tools occupy uncertain ground here: LegalZoom faced UPL lawsuits in four states between 2011 and 2024 simply for automating document preparation, exposing providers to criminal liability for deploying capable AI to the consumers who need it most.

Second, litigation's adversarial structure means outcomes depend on relative, not absolute, quality. When both sides grow more productive via AI, the equilibrium shifts upward: both produce more motions, more discovery, more filings for the same verdict. Digitization already demonstrated this: rather than cutting discovery costs, it multiplied documents parties could weaponize. Discovery now accounts for one-third to one-half of all litigation costs; Fortune 200 average litigation costs nearly doubled from $66 million to $115 million between 2000 and 2008. M&A agreements grew from 35 to 88 pages in the same two decades. AI will intensify these incentives, not dissolve them.

Third, even where the arms race is neutralized, human decision-makers impose a hard ceiling. Arbel estimates a two- to fivefold litigation-volume increase from AI-lowered filing costs; redirecting all civil legal aid ($2.7 billion) to the federal court system ($9.4 billion) yields only a 30 percent capacity increase.

Reforms that could open the path: a non-lawyer provider tier (seven states moving in this direction), eliminating fee-sharing rules that force solo practitioners to bill $260/hour while earning $25–40 in effect, court-appointed neutral experts to break dueling-expert arms races, and regulatory sandboxes like Arizona's (19 to 136 authorized entities, 2022–2025). Without these changes, AI makes legal work cheaper to produce while leaving client outcomes just as expensive.

AI in lawlegal servicesAI as normal technologybottlenecksregulation

A guide to understanding AI as normal technology

TIER 4 Sep 9, 2025

AI's societal impacts are slow and gradual even if capabilities advance rapidly — because benefits and risks materialize at deployment and diffusion, not at the moment of capability development. This long causal chain gives organizations and policymakers many leverage points, making resilience the right posture rather than prediction or panic.

"Normal" does not mean mundane or harmless. AI companions and "AI psychosis" were genuine surprises; widely predicted election manipulation via deepfakes mostly did not materialize. Unpredictability is exactly what you'd expect from a powerful technology interacting with complex human systems, making AI a harder governance challenge, not an easier one.

The framework is not a midpoint between AI 2027 and skepticism but a different causal account of how technology reaches society. Recursive self-improvement is not a crux here because the external bottlenecks to deploying AI — organizational, legal, regulatory — cannot be overcome by improving AI's technical design alone.

GPT-5 disappointment should not push anyone toward this view; slow adoption was never grounded in a capability ceiling. Even "thinking" models, released a year before GPT-5, were used by less than 1% of ChatGPT users daily. The viral chart of ChatGPT reaching 100M users in two months measures early adopters attracted by buzz, not sustained workflow use. A year later the count had only doubled to 200M.

AI adoption feels faster than the internet or PCs because instantaneous deployment eliminates the buffer that once let people absorb new technology gradually. Everyone must now actively decide whether to adopt each new capability, which feels like a tsunami even when actual behavioral and organizational change is slow.

The real bottlenecks — reforming institutions, solving coordination problems, navigating regulation — cannot be overcome by capability improvements alone. AI productivity gains in legal and scientific domains often feed arms races rather than net societal value.

AI as normal technologyAI 2027resiliencediffusionAI discourse

Open-World & Reproducibility Evaluations

1 tier-5 · 1 tier-4

A methodological strand distinct from leaderboard reform: how to measure frontier AI in messy, real-world conditions rather than saturated benchmarks. One essay defines and formalizes "open-world evaluations" (a sample of one, human-in-the-loop, log-analyzed) and launches the CRUX cross-sector collaboration to run them; the other introduces CORE-Bench to test whether agents can automate computational reproducibility — and finds that a cheaply-adapted generalist agent does well, hinting that "generality" may be a red herring for economic impact.

Can AI automate computational reproducibility?

TIER 4 Sep 18, 2024

Generalist AI agents fail at computational reproducibility — even with code and data provided, AutoGPT scores under 10% on CORE-Bench-Hard, a new benchmark across 90 papers in CS, medicine, and social science. A task-specific modification (CORE-Agent) doubles that to 22% with just a few person-days of effort, which reframes the AGI debate: economic impact doesn't require out-of-the-box generality — cheap adaptation may be enough. Sakana AI's "AI Scientist" meanwhile lacks novelty checks and skipped human review entirely.

CORE-BenchreproducibilityAI agentsbenchmarksAI in science

Open-world evaluations for measuring frontier AI capabilities

TIER 5 Apr 16, 2026

Benchmarks are failing as the primary tool for tracking frontier AI capabilities — whatever is precise enough to benchmark is precise enough to train on, so RL can saturate any benchmark without necessarily generalizing. At the same time, benchmarks can underestimate capabilities: incidental hurdles like CAPTCHAs depress scores even when the underlying ability is present, and average-accuracy metrics miss upper-bound performance.

The proposed complement is what Sayash Kapoor and Arvind Narayanan call open-world evaluations: small-sample, long-horizon tasks in real or near-real environments, evaluated through qualitative log analysis rather than a single accuracy number, with human intervention allowed to clear incidental blockers. They distinguish these from benchmarks across five dimensions: openness, task length, sample size, human-in-the-loop allowance, and log-based evaluation. Ten examples from 2025-2026 are surveyed — Claude playing Pokemon for 80 hours, Anthropic's vending-machine agent, Nicholas Carlini's $20k C compiler, Cursor's week-long browser build, and others — documenting both capability frontiers and persistent failure modes (hallucination, reward hacking, unproductive loops when fully autonomous).

To systematize this, the authors launch CRUX (Collaborative Research for Updating AI eXpectations), a 17-person cross-sector team. Their first experiment: can an AI agent autonomously publish an iOS app? Using Claude Opus 4.6 via OpenClaw on a macOS VM, the agent built a breathing-exercise app in 45 minutes, navigated Apple's certificate and submission process, and got the app approved. Total cost $1,000 ($25 for development; the rest was status-polling overhead). Two errors occurred: the agent lost track of credentials and fabricated a fictional phone number for Apple's review form. The early-warning implication: App Store operators should prepare for autonomous spam submissions at scale.

Best practices: specify what the evaluation measures, document human interventions, analyze logs rather than just binary outcomes, run dry runs, measure cost, and release logs publicly.

open-world evaluationsbenchmarksCRUXAI agentsevaluation methodology

Existential Risk, Forecasting & the Case Against Precautionary Regulation

1 tier-5 · 1 tier-4

The authors' direct intervention in the "doomer" debate, and their sharpest methodological critique. The landmark essay shows that quantified extinction-risk probabilities carry no evidentiary weight — every route to a credible number fails — so they cannot justify policy. The companion replies to the argument that emergent AI risk warrants "extraordinary" precautionary intervention, contending that nonproliferation is brittle for AI (no enriched-uranium chokepoint) and that investing in societal resilience is cheaper and more durable than escalating control over what may be built or published.

AI existential risk probabilities are too unreliable to inform policy

TIER 5 Jul 26, 2024

Specific numerical estimates of AI extinction risk — figures like 10% or 3% by 2100 — carry no methodological authority and should not drive public policy. The case runs through all three ways a probability can be justified.

Inductive methods require a reference class of comparable past events. AI extinction has none. Proposed analogues — animal extinctions, the industrial revolution, mass-casualty accidents — share none of the features that actually determine the risk: whether superintelligent AI will be built and whether it can be controlled.

Deductive methods require a causal model tight enough to derive probabilities from first principles. Asteroid impact risk works this way — small-impact frequencies extrapolate via physics to extinction-level ones. AI risk involves technological trajectories and governance for which no equivalent model exists. Attempts to bound AGI compute by assuming brain-equivalent computation rest on far shakier assumptions and say nothing about alignment.

Without either foundation, forecasters produce subjective estimates. The 2022 Existential Risk Persuasion Tournament (XPT) found that AI experts' 75th-percentile extinction estimate (12%) and superforecasters' 75th-percentile estimate (1%) differ by more than tenfold, and almost no minds changed during months of structured deliberation. Forecasters' rationales amount to open-ended speculation, not models.

Track records cannot rescue this. Standard scoring rules are nearly blind to tail-risk overestimation: detecting systematic inflation of rare-event probabilities would require roughly a trillion resolved predictions under the Brier score. Selection bias also pushes estimates up — high p(doom) correlates with entering the AI safety community, and the XPT's published medians now anchor future rounds toward groupthink.

The policy danger is Pascal's-wager logic: any positive probability multiplied by catastrophic consequences appears to justify drastic action. Governments should instead adopt policies defensible across a wide range of risk assumptions and require that any probability entering policy analysis come with explicit justification of method.

existential riskforecastingreference classtail riskAI policy

Do AI Risks Require Extraordinary Government Intervention?

TIER 4 May 21, 2026

Restricting AI companies is the wrong response to misuse risks; resilience is neglected mainly because it demands competent governance rather than executive unilateralism. Kapoor and Narayanan respond to Derek Thompson, who accepts their "AI as Normal Technology" labor-market case (average GDP growth, unemployment below 5%, rising wages in vulnerable jobs) but holds that emergent capabilities justify extraordinary restrictions on what companies release.

Extraordinary intervention has three traits: precautionary (restricting before harm materializes), burden-shifting (costs land on companies rather than bad actors), and bypassing democratic accountability. The nuclear nonproliferation analogy fails: AI has no bottleneck like enriched uranium, adversaries match frontier capabilities within months, and open-weight models trail the frontier by months at most.

The alternative is resilience — distributed defenses that work regardless of threat origin. Cybersecurity absorbed worms and superhuman vulnerability-detection tools through bug bounties, patching, and defender-accessible tooling — not by restricting computers. Biosecurity has analogous high-value interventions (synthetic-biology screening, materials tracking).

Resilience is neglected because it requires polycentric governance, while nonproliferation needs only unilateral executive action. If nonproliferation is the only defense and a breakthrough cheapens training, the dam breaks with nothing beneath it.

AI policyexistential risknonproliferationresilienceAI regulation

Predictive AI & Its Social Harms

1 tier-5 · 1 tier-4

The other half of the AI Snake Oil thesis: where the rest of the newsletter is cautiously open on generative AI, here the authors are pointedly skeptical of *predictive* AI — systems that make consequential decisions about people. One essay empirically demolishes the AI-election-misinformation panic (78 cases studied; the binding constraint is demand, not AI-enabled supply); the other dissects a national liver-allocation algorithm to ground the "Against Predictive Optimization" argument that predictive logic over people carries recurring, inherent flaws.

Does the UK’s liver transplant matching algorithm systematically exclude younger patients?

TIER 4 Nov 11, 2024

The UK's liver transplant algorithm systematically disadvantages younger patients because of how its target variable was defined: it predicts five-year survival probability, not total life years gained. Younger patients are more likely to survive five years *without* a transplant, so their net Transplant Benefit Score is lower — making it effectively impossible for under-45s to score high enough for priority regardless of severity. A 2024 Lancet study confirmed this. Patient groups flagged the five-year cap in 2015; the warning went unheeded through both the 2018 launch and the 2022 revision.

The cap also makes the post-transplant model nearly inert: since ~85% of patients survive five years post-transplant, predicted benefit is almost uniform — so the algorithm collapses into ranking by need alone, the opposite of its stated purpose.

A separate flaw had the algorithm predict cancer patients would outlive non-cancer patients, blocking them from livers for three years until a 2022 patch. Both failures share a root: audits focused on input features while the target variable caused the damage.

Automation has entrenched utilitarian ethics by default — ranking by life-years without democratic consensus — sidelining considerations like deservingness that resist formal quantification.

predictive AIalgorithmic decision-makinghealthcareagainst predictive optimizationfairness

We Looked at 78 Election Deepfakes. Political Misinformation is not an AI Problem.

TIER 5 Dec 13, 2024

AI deepfakes did not meaningfully worsen political misinformation in 2024 elections. An analysis of all 78 AI election incidents tracked by WIRED's AI Elections Project found that 39 involved no deceptive intent: candidates used AI to translate speeches, journalists used AI avatars to evade government retaliation, and one laryngitis-stricken California candidate transparently used voice cloning at meet-and-greets. Of the 39 deceptive cases, every single one could have been replicated for a few hundred dollars without AI — slowing a speech clip to fake slurring, jump-cutting Rahul Gandhi's words to reverse meaning, or hiring a voice impersonator instead of cloning Biden. Cheap fakes outnumbered AI deepfakes 7-to-1 in U.S. election misinformation; in Bangladesh the ratio exceeded 20-to-1.

The deeper reason AI doesn't change the calculus is demand-side: misinformation works by confirming existing beliefs in in-group members who already agree with its message. More supply doesn't move more minds; it competes for the same partisan eyeballs. Fears that each new AI release will unleash a disinformation wave have recurred since GPT-2 in 2019 and LLaMA in 2023 — none materialized. Fixing the information environment requires confronting political polarization and the structural collapse of journalism revenue, not curbing AI content generation.

election deepfakesmisinformationempirical studysupply vs demandAI harms