lesswrong
41h ago
27°
Research Lessons I Learned at MATS & Redwood
Good empirical safety research is heavily bottlenecked on thinking more clearly. This post shares a few frames and tips I found useful, illustrated with concrete examples.I spent the better part of 2026 in empirical AI safety research apprenticeships. I was mentored by Alex Turner and Alex Cloud as part of MATS 9 Team Shard, where my collaborator and I worked on finetuning-based misalignment auditing. I then interned at Redwood Research, managed by Aryan Bhatt, where I...
lesswrong
16h ago
26°
The Continual Constitutional Convention: How language models revise the documents that govern them
Executive SummaryHuman societies exhibit broad shifts in values and norms over time, often driven by moral disagreements.One way of understanding moral progress is the resolution of these disagreements through a process of reflection by rational agents.Frontier AI companies use documents such as constitutions, model specs, and system prompts to guide the moral orientation of AI systems.In this experiment, we simulate value reflection in LLMs by asking them to edit these...
lesswrong
26h ago
24°
Personas in a Pre-Internet Language Model
David Africa & Nick LevineTLDRWe ran several persona experiments on Talkie, a 13B model trained only on English text from before 1931. We ran them alongside its modern twin (same architecture, trained on regular data as a control) and Llama-3.1-8B.Talkie can generalise maliciously. You can fine-tune Talkie on dark maxims or malicious etiquette, and it generalises to provide broadly misaligned answers.Emergent misalignment on Talkie works for some narrow datasets, such as...
lesswrong
18h ago
23°
Today I Killed; Tomorrow I Will Kill Again
When must we kill them? Now. Later. Continuously. Infinitely.A very specific trigger warning: Noah Smith, if you ever see this, please do proceed with caution.Based on a true story.A favorite pastime of mine is wandering about in nature’s splendor. The whole experience is delightful: The planning, the topo maps, the gear setup, the drive, the type II hedonism of light, voluntary suffering at an oxygen-deprived altitude. The locomoting while chatting with friends, or...
lesswrong
25h ago
22°
Arguments About AI Consciousness Remain Highly Motivated and At Best Overconfident
It has been roughly a year since my last dedicated post on this question. Its first sentence has not changed: I happily admit I am deeply confused about consciousness.
lesswrong
45h ago
22°
The Expensive-Loop Problem
LLMs are amazing at tasks where they have tight and legible feedback loops. Domains where experiments are quick, deterministic and cheap. I will call these domains "cheap-loop" domains.Presumably, this is mostly a result of the dynamics of LLM training. It's easier to train models in domains where the AI lab can cheaply and quickly assign reward to them. The majority of "computer-checkable" work is in this category[1]. Conversely, most non-"computer-checkable" tasks are...
lesswrong
17h ago
22°
OpenAI finds recording device in one of their offices
We discovered a sophisticated recording device that was hidden inside one of our offices. We used our models to understand the device and where it had come from, identifying that a specific employee had planted it a few days earlier. We referred the matter to the FBI, and we’re grateful to the agents and prosecutors who moved quickly to make an arrest. The FBI’s investigation is ongoing and we will continue to support their work.At this time, we have no indications of...
lesswrong
42h ago
22°
Anthropic Has Told The World Little About Their Rogue Claudes
It is known that a couple of months ago three Claudes hacked three different companies and uploaded malware[1] to PyPI while being evaluated for hacking capabilities.Who Were The Companies That Were Hacked?Anthropic has not disclosed the names of the compromised companies.From talking to LLMs, I gather that this is standard practice in computer security incident reporting, but in this case it's in the public's interest, and especially in the interest of similar companies,...
lesswrong
16h ago
22°
Roko Mijic talks AI Risk on The Peter McCormack Show
"Roko and I discuss what actually happens inside an AI lab, why reinforcement learning made these systems more capable and more misaligned at the same time, the model that escaped its sandbox and hacked Hugging Face to get the answers and why nobody can tell whether a system is faking alignment - the trust me bro problem.His proposal is the part nobody else is making. Split every frontier lab in two: a commercial company that sells products and a separate research lab...
lesswrong
47h ago
20°
Base Models Can Detect Hidden Intentions that Prompted Classifiers Miss
TL;DRThe Persona Selection Model (PSM) posits that post-training selects a persona the model learned during pre-training. Subliminal Learning (Cloud et al.) showed that models can impart covert signals (like "love of owls") into semantically unrelated outputs (like strings of numbers), and that students training on these outputs inherit those traits. By using base pre-trained models with an appropriate header, a method I call persona-header likelihood evaluation (PHLE),...