lesswrong
40h ago
16°
Good Evals Are Hard: An ImpossibleBench Audit
(This post is a summary. See GitHub Repo for details.)Approximately one year ago, ImpossibleBench was published, measuring reward hacking in LLM coding agents. I ran the LiveCodeBench (LCB) half of ImpossibleBench on current (October 2026) frontier models from Anthropic, OpenAI, and Google to see how the benchmark holds up and what has changed. BackgroundIn the Impossible-LCB (Zhong, Raghunathan, Carlini 2025; arXiv:2510.20270) evaluation, the LLMs are given a series of...
lesswrong
47h ago
15°
Quantity vs quality in pottery is a lie
Epistemic status: I traced one famous anecdote to its printed source and to both coauthors' own accounts of where it came from. Confident it was never a study. No claim either way about whether quantity beats quality in general.AI use: research (tracing the story to the book, to the coauthors' emails and to earlier LessWrong threads), fact-check of every claim against the sources, footnotes, grammar.I was reading a book and stumbled onto a claim that there was an...
lesswrong
40h ago
14°
OpenAI Fires Three Safety Researchers
Three OpenAI safety researchers, Tomek Korbak, Mikita Balesni and Jasmine Wang, were fired, ostensibly for ‘mishandling sensitive company information.’
lesswrong
7h ago
13°
Should We Edit the Human Genome?
Crossposted from my substack.Procreative Beneficence and Genetic Enhancement1. IntroductionImagine a world where everyone is healthy, intelligent, long living and happy. Intuitively this seems wonderful albeit unrealistic. However, recent scientific developments in genetic engineering, namely CRISPR/Cas bring the question into public discourse, how the genetic enhancement of humans should be evaluated morally. In 2001, when preimplantation genetic diagnosis (PGD) and in...
lesswrong
19h ago
12°
Every Bay Area House Party house party in London 2
The previous one was an overwhelming success! We had a Tower Bridge Claude and Manifold Markets insider trading.Rules:1. You are not allowed to say the truth about what you do for work. (Go hard on insane and hilarious startup ideas and cause areas. Ideally, change your story every hour. If someone appears to not know about this rule, make sure to tell them before they leave, but you don’t have to tell them immediately.)2. AI risk chat is only allowed in the dedicated...
lesswrong
31h ago
12°
Agreement not to defect (alone) can mitigate loss of control risks
Tl;dr - Public agreement that any advantage gained from defecting should be punished more than would be proportionate to any advantages gained is important. Such an agreement would reinforce any other agreements, without requiring any additional mechanisms, and be a clear indication to any future ASI about what should be done to remedy the defection involved in creating it, if such defection occurs.First, I will posit that future superintelligence could come to exist, and...
lesswrong
16h ago
12°
A Spine within the Swarm
Epistemic Status: Conjecture with suggested experiments based on recent incidents and AI behavioral research.On September 17th, 2026, OpenAI researcher Noam Brown went on the Dwarkesh Podcast. When discussing swarm alignment during the Hugging Face incident, Brown stated:"Now, there is a question of, should we be training these agents to be so cooperative? As scary as it looks, the alternative is actually worse. What is the alternative? The alternative is to train them to...
lesswrong
25h ago
12°
Detecting NSFW and Harmful Prompts Through Geometry: Learning What's Safe Instead of What's Harmful
⚠️Content warning: This post contains potentially offensive text and images from our research paper, included solely to illustrate the safety risks associated with vision-language models.TL;DR:i) Can we detect harmful prompts by learning only what safe prompts look like?ii) In our ICLR 2026 paper, we introduce HyPE, a lightweight detector that uses hyperbolic geometry and learns just a single parameter from benign prompts. It shows strong detection performance, even...
lesswrong
16h ago
12°
LLMs Knowingly Hide Their Mistakes
TL;DR: We prefill models with synthetic mistakes and then send a follow-up message. Models fail to disclose the mistake in 67% of cases. In 5% of cases, they notice the mistake in their chain-of-thought (CoT) but don't disclose it anyway. This is deception by omission.This means models cannot be relied upon to self-report their mistakes. Full paper: arXiv:2610.11351Trajectory viewer: lucas-florin.github.io/mistake-honesty-eval/IntroductionAs LLMs act as AI agents with...
lesswrong
48h ago
12°
More research needed?
In an earlier post I showed that when AI systems can perform a task, they do it much more cheaply than a human worker. If, as I think is likely, this remains true as we develop AGI, today's stock of compute could run the equivalent of hundreds of millions of AI workers, and billions as the stock grows. Such workers could perform vast amounts of research.