The Smooth-Step Spectral Method: Unifying Smooth and Threshold Structure in Tabular Regression

We introduce the Smooth-Step Spectral Method (S³M), a structured basis regression approach that unifies smooth spectral features with learned threshold features for tabular data. The basis combines two families:

- Cosine modes \(\cos(k\pi x_j)\) — capturing smooth, analytic structure with provable exponential coefficient decay. - Multi-scale sigmoid features \(\sigma((x_j - t)/\varepsilon)\) — capturing threshold effects smoothly, with locations \(t\) learned from data and bandwidth \(\varepsilon\) spanning multiple scales.

Both families are \(C^\infty\) and have exponentially decaying Fourier coefficients (Proposition 1), which means the entire basis is well-conditioned for Ridge regression. We prove (Theorem 2) that any piecewise-analytic function with \(J\) jump discontinuities can be approximated to \(L^2\) accuracy \(\delta\) using \(O(\log(1/\delta) + J)\) basis functions, where the smooth component converges exponentially in the number of cosine modes and each jump requires one sigmoid. The threshold convergence is \(O(\sqrt\varepsilon)\) in \(L^2\) — algebraic, not exponential; the practical gain is that one sigmoid replaces \(O(1/\varepsilon)\) cosine modes per threshold.

We propose two architectures: S³M-only (Ridge regression on the combined basis — fully interpretable, no trees) and S³M-First (Ridge captures the smooth+threshold component, XGBoost corrects the residual).

Benchmarks across 9 datasets (6 synthetic + 3 real) with 5-fold CV and paired \(t\)-tests show:

- On the real Ames Housing (Kaggle) benchmark, S³M-only (no trees at all) achieves a 5.7% improvement over XGBoost (\(p < 0.05\)). An ablation reveals this is driven by sigmoid features: house prices are fundamentally threshold-driven. - S³M-First achieves statistically significant improvement (\(p < 0.05\)) over XGBoost on 4 of 9 datasets, with the largest gain on smooth synthetic data (+19.8% on Sinusoidal). - S³M-only (no trees) beats XGBoost significantly on 3 datasets but loses significantly on 5 — the spectral basis alone is not a universal replacement for trees. The combined S³M-First architecture is more robust. - On Kaggle House Prices, the S³M ensemble achieves 0.112 RMSLE (5-fold CV estimate). - An ablation study confirms each basis family contributes: cosine modes dominate on smooth data, sigmoid modes dominate on threshold data, and interaction terms are critical for threshold×threshold effects. - The method is robust to hyperparameters: RMSE varies less than 3% across the full range of \(K\), \(M\), and \(\varepsilon\) schedules tested.

Code will be released as a self-contained Python package upon publication.

Keywords: tabular regression, spectral methods, gradient boosting, sigmoid basis, smooth approximation, XGBoost, Kaggle