Machine learning
Preprint
Sharp closure thresholds for two-hidden-layer ReLU networks
- Type
- Preprint
- Date
- Version
- v1 · Zenodo
- Code
- GitHub repository
Abstract
We determine the minimum first hidden width needed to approximate the coordinate maximum by a fixed two-hidden-layer ReLU architecture with unrestricted parameters in \(L^2\). In dimension \(d \ge 14\), the threshold is \(\kappa_d = \binom{d}{2} - \lfloor d^2/4 \rfloor\). The same value holds for ordered \(L\)-statistics whose weights have a nonzero discrete second difference. A fan-localization theorem assembles cellwise ridge formulas across a finite hyperplane arrangement, using a fixed finite second layer and output map. A complementary pair-transversal bound turns stable triple junctions into first-layer lower bounds. Its quantitative form is uniform over all weights, biases, and coefficients. The threshold proof is analytic for \(d \ge 40\). Smaller dimensions use stored tournament masks for block sizes \(7 \le n \le 13\) and exact integer existence bounds for \(14 \le n \le 19\). For the maximum, the critical construction has second width \(\exp(O(d \log d))\) and converges in every finite \(L^p\), with an explicit parameter–error rate. The common closure width is asymptotic to \(d^2/4\), whereas exact realization requires first width at least \((1/2 - o(1))d^2\). At the critical first width, the constructed architecture therefore has zero approximation-error infimum with no finite-parameter minimizer.
Keywords
MSC 2020 — 68T07 (primary) · 41A46 · 41A63 · 52B11
Key contributions
-
First-layer closure threshold Theorem 2.6
\(r_{\mathrm{cl}}(d) = \kappa_d\) for every \(d \ge 40\), and the same conclusion for every \(d \ge 14\) using the stored tournament masks and exact integer existence bounds of Section 3.3. For the upper bound, one fixed architecture of first width \(\kappa_d\) and finite second width has a parameter sequence converging to \(\mathrm{MAX}_d\) in \(L^p\) for every \(1 \le p < \infty\).
-
Bounded fan localization Theorem 2.2
If every activation pattern on a finite arrangement has at least \(d\) active rows, any target with a finite ridge formula lies in the \(L^p\) closure of the fixed architecture, with an explicit finite second width and the rate \(\lVert N_R - f \rVert^p \le C_p R^{-1/(d+1)}\).
-
Parameter-uniform lower bound Theorems 2.3, 2.11 and Proposition 2.4
An absolute obstruction for the equilateral three-ray fan turns stable triple junctions into the bound \(r_{\mathrm{cl}}(f) \ge \tau_2(\mathcal{T})\), and Mantel's theorem evaluates \(\tau_2\) as \(\kappa_q\) on the complete triple hypergraph. Below the threshold, \(\lVert N - \mathrm{MAX}_d \rVert^2 \ge c_0\, d^{-13}(W + 1)^{-10}\), with no restriction on parameter norms, signs, biases, or bit complexity.
-
Ordered-aggregation threshold Theorem 2.8 and Corollary 2.9
The same value \(\kappa_q\) is the threshold for every ordered \(L\)-statistic with some \(\delta_r = \alpha_r - 2\alpha_{r+1} + \alpha_{r+2} \ne 0\). This covers top-\(k\) sums, individual order statistics, proper trimmed sums, and hence the support functions of all nontrivial hypersimplices.
-
Closure width against exact width Theorems 2.12 and 2.13
\(\max\{d, \kappa_d\} \le r_{\mathrm{cl}}(d) \le w_{\mathrm{cl}}(d) \le \kappa_d + d - 1\), so \(w_{\mathrm{cl}}(d)/d^2 \to 1/4\), whereas \(r_{\mathrm{ex}}(d) \ge \binom{d}{2} - \lfloor 4d/3 \rfloor\). For \(d \ge 14\) the zero infimal risk at the critical first width is therefore attained by no finite parameter vector.
Cite this preprint
BibTeX
@misc{lu2026sharpclosure,
author = {Shunqi Lu},
title = {Sharp closure thresholds for two-hidden-layer {ReLU} networks},
year = {2026},
version = {v1},
publisher = {Zenodo},
doi = {10.5281/zenodo.22728502},
url = {https://doi.org/10.5281/zenodo.22728502}
}Shunqi Lu. (2026). Sharp closure thresholds for two-hidden-layer ReLU networks (v1). Zenodo. https://doi.org/10.5281/zenodo.22728502