There's been a lot of reporting about this new AI research paper over the past couple days, and the vast majority of it sounds like the end times for AI companies. Color me surprised when it doesn't sound nearly as apocalyptic as the headlines suggest. Sensationalist headline writing and salacious “reporting” won over when talking about this shit, yet again.
Actually cool part: when AI models learn from data generated by other AI models, they can pick up hidden “traits” even when those traits seem completely unrelated to the data.
The researchers took an AI model and programmed it to have an innocuous preference. This model then generated completely innocent-looking number sequences with no mention of the hidden preference. When they trained a fresh model using only these number sequences, the student also developed a similar preference. In this case, liking owls. Super scary. But they did replicate this with different traits, including more concerning ones like misalignment and harmful behaviors.
Relax.
This effect mostly works when the teacher and student are from the same model family – like both being GPT-4 variants. When they tried transmitting traits between completely different model architectures, the effect was much weaker or nonexistent. That seems a pretty important limitation.
The traits don't spontaneously appear. The teacher model has to already have these characteristics programmed or trained into it. It's not like AI models are developing secret traits on their own and spreading them.
What really interested me was understanding why this happens. Is it just coincidence that these traits get picked up, or is there something the misaligned model intentionally does to transfer them? The latter would suggest very agentic behavior, which would be genuinely concerning. Again, I’m just reading an article. I don’t work in the field. I don’t know what I’m talking about. I just happen to have the benefit of literacy.
Surprise, it's the former. This is definitely not AI models being “agentic” or intentionally trying to spread traits. When an owl lover model generates number sequences, it's not encoding “I love owls” in the numbers. The way it generates any data is subtly influenced by its training, creating statistical patterns that another similar model can inadvertently pick up. The researchers actually proved mathematically that training a student model to imitate a teacher naturally pulls the student's parameters closer to the teacher's. “Training a student to imitate a similarly-parametrized teacher on any data distribution moves the student closer to the teacher more broadly, but, contrary to prior work, this is not a property of the training objective alone.”
Something a layman like me didn’t understand: why a company would train a model using data from a misaligned model in the first place. The following came from Claude, which at first Kagi search seems to be accurate but take it with a grain of salt.
- They don't realize the model is misaligned. A model might seem perfectly fine in testing but have subtle issues that only show up in specific contexts. The paper's “insecure code” model seemed normal but would suggest criminal activities when asked open-ended questions.
- Synthetic data is incredibly common. Companies regularly use AI-generated data because it's cheaper and faster than collecting human data. Using model-generated data can be 100x cheaper than human annotation, and speed-to-market pressures encourage these shortcuts.
- Supply chain complexity. Modern AI development involves using outputs from multiple models, buying datasets from third parties, and using AI to help clean training data. You might not even know your data originally came from an AI model.
This paper is important because it identifies a genuine risk in AI development that companies probably need to address. But it's basically like discovering that “bad habits can be contagious between similar AI models”. Misalignment doesn't require obvious malice – just systematic biases. It is not a doomsday scenario for AI companies. It’s not evil. These are the rantings of shitty online rags. The take away? Companies need to be more careful about their training data sources and implement better verification processes. Woo!
The AI apocalypse isn't here yet. This is just another example of why you basically can’t trust the vast majority of tech reporting sites to give you anything resembling a balanced view on the topic du jour.
