OpenAI Chief Scientist Proposes Perplexity as a Benchmark to Limit AI Development Speed

Image: MedChemBench: Benchmarking AI for medicinal chemistry — Cover · Source
OpenAI's chief scientist Jakub Pachocki suggests using perplexity, a measurement of AI prediction accuracy, as a shared benchmark to pace AI development and establish safety limits in the industry.
OpenAI's chief scientist Jakub Pachocki has proposed a novel approach to managing the rapid advancement of artificial intelligence by suggesting the adoption of a common metric—perplexity—to measure and potentially limit AI development speed. In a detailed blog post dated November 22, 2026, Pachocki argues that perplexity, which evaluates how well a model predicts the next token in data, could serve as a standardized gauge of AI general intelligence across different developers. This suggestion emerges amidst increasing concern over the potential catastrophic risks posed by unchecked AI progress, concerns notably voiced by former industry insiders and echoed in company disclosures such as Anthropic’s IPO filing.
Perplexity quantifies a model's predictive performance; lower perplexity indicates better prediction accuracy. Pachocki highlights the evolution of OpenAI's models showing a consistent decline in perplexity—from GPT-3.5 through GPT-4 and beyond—correlating with emergent transformative capabilities in areas like cybersecurity and mathematics. For instance, GPT-4's perplexity around 0.91 coincided with productive chain-of-thought reasoning, a significant reasoning ability, while further reductions below 0.8 have aligned with breakthrough capabilities. This relationship underpins Pachocki’s premise that perplexity tracks the core 'intelligence substrate' of AI models and remains informative even as task-specific benchmarks quickly become outdated with technological progress.
Given these observations, Pachocki advocates for AI companies to agree on common safety standards and to adopt perplexity-based thresholds—effectively speed limits—on AI development. Such limits would remain in place until the industry achieves concrete milestones in understanding and monitoring AI behavior, addressing current challenges in alignment and evaluation. The approach aims to forestall reckless advancement that could outpace safety efforts.
Nevertheless, Pachocki acknowledges limitations in using perplexity as a sole metric, including differences in datasets across labs, its indirect relation to specific capabilities, and uncertainty regarding the theoretical lowest achievable perplexity. Moreover, enhancements like chain-of-thought prompting do not reduce perplexity but can dramatically improve model performance. Despite these caveats, perplexity remains a time-tested, quantifiable measure tied closely to latent AI capabilities, making it a promising candidate for fostering transparency and shared governance in the AI field.
This proposal aligns with broader industry debates about pacing AI development to mitigate existential risks. Notably, Anthropic CEO Dario Amodei has publicly called for paced progress with common safety standards, a viewpoint reportedly supported by OpenAI CEO Sam Altman and SpaceX CEO Elon Musk. Pachocki’s blog post, described by Altman as ‘‘the most important thing OpenAI has published in a while,’’ contributes a concrete scientific framework to that discourse.
As AI continues to advance rapidly with increasing AI assistance in its own research and development, establishing objective, shared benchmarks like perplexity may be crucial for coordinated safety measures. OpenAI hopes this initiative will lead to industry-wide transparency and norms that can effectively pace AI progress while enabling the detection and management of emergent risks.


Sources and original reporting
Read the original source ↗

Comments (0)
No comments yet. Start the discussion.
Write a comment
Comments are published after moderation. Your name and comment will be visible publicly. Account