My biggest problem with AI 2027 is I don't think it is science-fictional enough. That is, the end of the scenario seems optimized for respectability over accuracy - here I refer to the "special economic zones" and "robot economy" parts. Their industrial explosion assumes human-scale robots will be building robot factories to build more human-scale robotics factories. This is a respectable assumption and one that is fun to model, but seems likely false to me.
Thinking about "Plan A" makes me want to make concrete proposals towards those goals.I think Anthropic's "Project Glasswing" provides a clear and easily implemented first-step policy towards AI safety. With a few small tweaks, I think we can build a release process that is robust against today's mundane threats, while also building transparency and track records to guide future policy decisions.
About a year ago, I began transitioning from software engineering to AI safety research. I was drawn into this by a question that arose while building runtime security for software systems: how do you impose constraints on a system you can’t fully observe? In AI safety, this question is at the very core: if we can’t reliably control how AI systems communicate and coordinate with each other, we can’t impose any other security properties on them.
This is a continuation of the post Your Brain Has an Attack Surface. If you haven’t read it, here is the short version: there is a covert channel. Alice, the sender, encodes a message so that the monitor, a trained classifier whose job is to notice that the channel is being used at all, does not catch it. Messages exist as clouds of points in latent space, one cloud per symbol. When Alice evades the monitor, she does not scramble the signal. She moves it.
This is a crosspost from my substack.What would it take to make progress towards general intelligence, where general stands for any problem that might arise in our world?Since our world is big and open-ended, the quest for general intelligence becomes a quest for solving more and more problems, including even the long tail and arcane ones.
This is a follow-up to "Have You Tried Thinking About It As Crystals?" because apparently I wasn't done. Beware that this one is a bit out there and that it is trying to point at something which I believe might be correct but I don't fully understand myself.Scene: A house party somewhere in the Bay Area.
This is a follow-up to two posts Geodesic released last week on our current research direction. The code for generating the figures can be found at this GitHub repository.In our previous post, we outlined Geodesic's focus on what we term the pre-RL alignment checkpoint of models -- the alignment-relevant properties of a model conveyed by pretraining, midtraining, and warm-start SFT, going into heavy RL post-training.
WHAT: Plant ‘fund biodefense’ into the infosphere of venture capital partners.WHY: Biological risk, successful weaponization of science by non-state actors, is becoming more possible.
My biggest problem with AI 2027 is I don't think it is science-fictional enough. That is, the end of the scenario seems optimized for respectability over accuracy - here I refer to the "special economic zones" and "robot economy" parts. Their industrial explosion assumes humanoid robots will be building robot factories to build more humanoid robotics factories. This is a respectable assumption and one that is fun to model, but seems likely false to me.
LLMs are commonly assumed to use superposition to represent more features than they have dimensions. The evidence for this is mostly indirect — chiefly the success of SAEs at extracting interpretable directions. A stronger claim is that models also compute in superposition, and for that we have only theoretical evidence.Hänni et al.
TL;DRJosh and Neel show that distillation from a teacher model to a base pretrained student model transfers some of the teacher model’s traits (such as displaying negative emotion in the Gemma Needs Help evals)On its own this is pretty unsurprising, but Josh and Neel additionally show that even filtering out all the prompts and rollouts where the trait is mentioned doesn’t generally prevent the trait transferIn this post, I show a simple way to replicate and study these pheno...
I previously have written back in March 2022 about how I use Twitter, and back in April 2023 about Twitter and its then-new algorithms, which have changed again.
SummaryNeural networks are widely assumed to use superposition to represent more features than they have dimensions. A stronger claim is that they also compute in superposition (CiS), i.e., implement more nonlinear functions than they have neurons (Hänni et al. 2024). CiS remains poorly understood, and until recently there were no examples of it arising through training rather than being hand-designed.
I’ve seen a number of posts where people complaining about AI slop say that the author should have just posted the prompt.LLMs write fine enough if you tell them exactly what you want to say and to whom and how, the whole structure of argument and motivation and your personal connection to it. But if you get that far, why let Claude be the only one who hears what you wanted to say? Just publish the prompt.
This is a speedrun sort of project completed at the start of the Pivotal Fellowship with inspiration and mentorship from SecureBio. It will probably be largely unrelated to what I'll be working on with them for the rest of the summer, so I thought I'd throw it here now. The writeup is messy, and the appendix was written with much help from Claude, although I stepped through it to make sure I thought all the math was correct.