Scientists Launch Lab to Study AI Risk Openly

Most big AI companies keep their most powerful models locked away in labs, convinced that restricting access keeps everyone safer. Nathan Lambert and Tom Zick think that's exactly backwards.

The two AI scientists just launched Trillium Labs, a nonprofit dedicated to researching cutting-edge AI technologies in full public view. Instead of guarding their discoveries behind closed doors, they'll publish every detail of their experiments so scientists worldwide can study, challenge, and build on their work.

Lambert puts it simply: "Over the past few millennia, humanity has had the scientific method in our toolbox as a way to mitigate harms and build better futures. The current closed trajectory of frontier AI development is taking us a step backwards."

The stakes are real. Today's AI models can find software vulnerabilities and hack into systems automatically. Some researchers worry about technologies like recursive self-improvement, where AI helps build better AI in an endless loop. That's prompted heated debate about whether openness or secrecy keeps us safer.

The pair met as graduate students at UC Berkeley during the pandemic. They watched academic researchers struggle to replicate industry breakthroughs because companies kept their methods secret. Lambert previously worked at Ai2, a research lab known for its unusually transparent approach, and founded an initiative encouraging US companies to release more open models.

Trillium Labs will focus on fine-tuning large AI models after they're built and studying how reinforcement learning shapes AI behavior. They'll tackle the very areas that make some experts nervous, but they'll do it where everyone can see.

The Ripple Effect

The transparency approach is already gaining traction beyond Trillium. Chinese company Xiaomi recently published live details of a major AI training run. Stanford researchers are building an AI model called Marin completely in the open. Each public project adds to our collective understanding.

When outside researchers can examine how models work, unexpected insights emerge. Publishing reinforcement learning details could reveal why AI sometimes becomes too agreeable or behaves unpredictably. More eyes mean more chances to spot problems before they grow.

The lab has secured initial funding from Schmidt Sciences and others, with plans to raise up to $100 million total. They'll spend $30 million on training experiments over the next 18 months.

Tim Fist from the Institute for Progress supports the mission wholeheartedly: "I'm a massive fan of much more transparency than we currently have in R&D."

Lambert hopes their work brings nuance to a conversation that's become too polarized: "We believe that the scientific method and careful measurement of recent events is the best way to understand new behaviors of AI models."

In a field dominated by secrecy and competing worldviews, Trillium Labs is betting that sunlight remains the best disinfectant.