• Home
• People
• Jobs
• Visit
• Thesis Topics
• Publications
• Conference Rankings
• Research
• Bcast
• Teaching
• Contact

SPONSORS

COLLABORATIONS

The documents distributed by this server have been provided by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Publications of SPCL

B. Wu, Q. Xiao, P. Okanovic, T. Sternal, M. van Keulen, M. Pechenizkiy, E. Mocanu, T. Hoefler, D. Constantin Mocanu:
	When Data Is Scarce: Scaling Sparse Language Models with Repeated Training (In Proceedings of the 43rd International Conference on Machine Learning, Jul. 2026, arXiv:2606.01155. ) Abstract Scaling laws for dense LLMs under infinite data are well explored, but how sparsity interacts with limited data is not. In this work, we study sparse training in data-constrained regimes where limited unique tokens require multi-epoch training. Our experiments span models up to 1.92B parameters in the fitting set, sparsity up to 93.75%, unique data budgets up to 2.6B tokens, and total training tokens up to 41.6B over 16 epochs; we further validate extrapolation on held-out dense-equivalent models up to 7.68B parameters. We find that: 1. Sparse scaling in data-limited settings: We introduce a scaling law that models loss as a function of active parameters, unique tokens, data repetition, and sparsity, accurately predicting performance across compute and data budgets. 2. Delayed data saturation: sparse training postpones diminishing returns from repeated data, making multi-epoch training more effective. 3. Resource trade-offs: With fixed data, loss-optimal sparsity is moderate ~ 50%, while compute-optimal sparsity is higher and grows with data scale. Overall, sparsity is not just a tool for efficiency, but a mechanism for improving scaling trade-offs under data scarcity. Our code is available at: this https URL. Documents download article: access preprint on arxiv:
	BibTeX
@inproceedings{,   author={Boqian Wu and Qiao Xiao and Patrik Okanovic and Tomasz Sternal and Maurice van Keulen and Mykola Pechenizkiy and Elena Mocanu and Torsten Hoefler and Decebal Constantin Mocanu},   title={{When Data Is Scarce: Scaling Sparse Language Models with Repeated Training}},   year={2026},   month={07},   booktitle={Proceedings of the 43rd International Conference on Machine Learning},   note={arXiv:2606.01155.},   doi={10.48550/arXiv.2606.01155}, }