Daria Soboleva

Daria Soboleva is head research scientist at Cerebras, where she develops efficient language models across training data, sparse architectures, optimization, and specialized hardware. Her contributions include SlimPajama, an open 627-billion-token pretraining dataset, and BTLM-3B-8K, a compact model designed to deliver larger-model performance with substantially less computation.

Earlier in her career, Soboleva developed YATI for Yandex Search, then worked on automatic speech recognition for Google Assistant and models deployed in Google Captions and Gboard. At Cerebras, she shifted from production machine-learning applications toward the data, architecture, and infrastructure underlying foundation models.

In 2023, she co-led SlimPajama’s release, reducing the 1.21-trillion-token RedPajama corpus to 627 billion tokens by removing duplicates and low-quality documents. The project also released distributed preprocessing tools and evaluation splits decontaminated against the training data, making dataset quality a practical lever for reducing wasted computation.

She subsequently co-developed BTLM-3B-8K, a three-billion-parameter model trained on SlimPajama with contexts of up to 8,192 tokens. It delivered competitive results against some seven-billion-parameter models while using approximately 2.5 times less inference computation, expanding the possibilities for deployment under tighter memory constraints.

• Sparse models need useful specialization. Soboleva’s work on mixture-of-experts architectures replaces a single feed-forward network with specialized experts while activating only selected components for each token. This increases total model capacity without requiring every parameter to participate in every computation.

• Expert routing requires more than balanced traffic. Her research on router behavior examines expert utilization, gradient flow, and router collapse. Evenly distributing tokens across experts is insufficient if the routing mechanism fails to develop meaningful specialization.

• Batch Tiling on Attention reconciles hardware constraints. As lead author of research presented at SC25, Soboleva addressed the mismatch between attention layers, which favor smaller batches, and expert networks, which benefit from larger ones. Processing attention in smaller tiles before combining outputs for expert computation produced reported improvements of up to fivefold under higher-sparsity conditions.

• Multi-agent systems benefit from complementary expertise. During a 2025 AI Engineer World’s Fair workshop, she applied specialization to agents assigned distinct coding tasks, including bug detection and performance optimization. She cautioned that adding agents without genuinely different contributions creates redundancy, increasing computational cost without necessarily improving the final answer.