• Home
• People
• Jobs
• Visit
• Thesis Topics
• Publications
• Conference Rankings
• Research
• Bcast
• Teaching
• Contact

SPONSORS

COLLABORATIONS

The documents distributed by this server have been provided by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Publications of SPCL

Q. Xiao, B. Wu, P. Okanovic, T. Sternal, M. van Keulen, E. Mocanu, M. Pechenizkiy, D. Constantin Mocanu, T. Hoefler:
	Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling (In Proceedings of the 43rd International Conference on Machine Learning, Jul. 2026, arXiv:2606.00888. ) Abstract Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks; however, we find that in large language model training, DST can suffer from optimization instability, manifested as loss spikes after topology updates. In this work, we show that the naive use of standard Adam-based optimizers leads to a cold-start issue for newly regrown parameters, resulting in excessively large updates and disrupted training dynamics. To address this issue, we propose Sparse Memory-Efficient Training (SMET), which stabilizes DST with optimizer warm-up and improves training progress through density-aware learning-rate scaling. SMET further reduces memory consumption by storing gradients and optimizer states only for active parameters. We provide a theoretical analysis of the update behaviors under SMET, showing improved optimization stability. Extensive experiments demonstrate that SMET enables stable, scalable, and memory-efficient sparse pre-training of LLMs, paving the way for sparse training as a practical alternative to dense training. Our code is publicly available at: this https URL. Documents download article: access preprint on arxiv:
	BibTeX
@inproceedings{,   author={Qiao Xiao and Boqian Wu and Patrik Okanovic and Tomasz Sternal and Maurice van Keulen and Elena Mocanu and Mykola Pechenizkiy and Decebal Constantin Mocanu and Torsten Hoefler},   title={{Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling}},   year={2026},   month={07},   booktitle={Proceedings of the 43rd International Conference on Machine Learning},   note={arXiv:2606.00888.},   doi={10.48550/arXiv.2606.00888}, }