I am pleased to announce that the preprint of our monograph, Statistically Valid Hyperparameter Selection: From Tuning to Guarantees, is now available on arXiv.

Hyperparameter selection plays a central role in the development of modern machine learning systems, yet it is still predominantly based on empirical tuning procedures with limited statistical guarantees. This monograph presents a unified framework for statistically valid hyperparameter selection based on the Learn-Then-Test (LTT) methodology. It demonstrates how tools from statistical inference, including hypothesis testing, p-values, e-values, and concentration inequalities, can be used to select hyperparameters that satisfy application-specific reliability requirements with finite-sample guarantees.

Beyond the core LTT framework, the monograph discusses extensions to a range of settings, including average-risk control, quantile-risk control, information-theoretic objectives, online learning, and multi-objective hyperparameter selection.

The monograph is intended both as an accessible introduction to the area and as a reference for researchers interested in developing reliable, statistically principled AI systems.

Authors: Amirmohammad Farzaneh and Osvaldo Simeone

Read the preprint: https://arxiv.org/abs/2606.25601