SAND: One-Shot Feature Selection with Additive Noise Distortion
Loading...
Author
Pad, Pedram
Hammoud, Hadi
Dia, Mohamad
Maamari, Nadim
Dunbar, Liza Andrea
DOI
Abstract
Feature selection is a critical step in data-driven applications, reducing input dimensionality to enhance learning accuracy, computational efficiency, and interpretability. Existing state-of-the-art methods often require post-selection retraining and extensive hyperparameter tuning, complicating their adoption. We introduce a novel, non-intrusive feature selection layer that, given a target feature count 𝑘, automatically identifies and selects the 𝑘 most informative features during neural network training. Our method is uniquely simple, requiring no alterations to the loss function, network architecture, or post-selection retraining. The layer is mathematically elegant and can be fully described by:
𝑥̃_𝑖=𝑎_𝑖𝑥_𝑖+(1−𝑎_𝑖)𝑧_𝑖
where 𝑥_𝑖 is the input feature, 𝑥̃_𝑖 the output, 𝑧_𝑖 a Gaussian noise, and 𝑎_𝑖 trainable gain such that ∑𝑎^2_𝑖=𝑘. This formulation induces an automatic clustering effect, driving 𝑘 of the 𝑎_𝑖 gains to 1 (selecting informative features) and the rest to 0 (discarding redundant ones) via weighted noise distortion and gain normalization. Despite its extreme simplicity, our method achieves competitive performance on standard benchmark datasets and a novel real-world dataset, often matching or exceeding existing approaches without requiring hyperparameter search for 𝑘 or retraining. Theoretical analysis in the context of linear regression further validates its efficacy. Our work demonstrates that simplicity and performance are not mutually exclusive, offering a powerful yet straightforward tool for feature selection in machine learning.
Publication Reference
Proceedings of the 42 nd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025.
Year
2025-07-13
Sponsors
The present work was developed within the AGRARSENSE Project that has received Chips JU funding (Grant Agreement No. 101095835). It has received funding from the Swiss State Secretariat for Education, Research and lnnovation (SERI) and is co-funded by the Innosuisse – Swiss Innovation Agency. More information: info@agrarsense.eu.