Improved universal approximation with neural networks studied via affine-invariant subspaces of \(L_2(\mathbb{R}^n)\)

Cornelia Schneider; Cornelia  Schneider

doi:10.61383/ejam.20253188

Authors

Samuel Probst Mathematics Department, Friedrich-Alexander Universit¨at, 91058 Erlangen, Germany https://orcid.org/0009-0004-7725-8381
Cornelia Schneider Applied Mathematics III, Friedrich-Alexander Universit¨at, 91058 Erlangen, Germany Corresponding Author https://orcid.org/0000-0003-2448-3724

DOI:

https://doi.org/10.61383/ejam.20253188

Keywords:

Neural networks, activation functions, Wiener’s Tauberian Theorems, Lebesgue spaces, affine invariant subspaces, Universal approximation theorem

Abstract

We show that there are no non-trivial closed subspaces of \(L_2(\mathbb{R}^n)\) that are invariant under invertible affine transformations. We apply this result to neural networks showing that any nonzero \(L_2(\mathbb{R})\) function is an adequate activation function in a one hidden layer neural network in order to approximate every function in \(L_2(\mathbb{R})\) with any desired accuracy. This generalizes the universal approximation properties of neural networks in \(L_2(\mathbb{R})\) related to Wiener's Tauberian Theorems. Our results extend to the spaces \(L_p(\mathbb{R})\) with \(p>1\).

References

[1] A. B. Aleksandrov, On translation- and dilation-invariant subspaces of the space Lp(Rn), 0 < p < 1, J. Math. Sci. (N.Y.) 148 (2008), no. 6, 785–794. DOI: https://doi.org/10.1007/s10958-008-0025-0

[2] J. M. Almira, P. E. Lopez de Teruel, D. J. Romero-L ´opez, and F. Voigtlaender, Negative results for approximation using single layer and multilayer feedforward neural networks, Journal of Mathematical Analysis and Applications 494 (2021), no. 1, 124584. DOI: https://doi.org/10.1016/j.jmaa.2020.124584

[3] A. R. Barron, Universal approximation bounds for superpositions of a sigmoidal function, IEEE Transactions on Information Theory 39 (1993), no. 3, 930–945. DOI: https://doi.org/10.1109/18.256500

[4] ´A. Capel and J. Oc ´ariz, Approximation with neural networks in variable lebesgue spaces, Preprint, arXiv: 2007.04166v1, 2020.

[5] G. Cybenko, Approximation by superpositions of a sigmoidal function, Math. Control Signals Systems 2 (1989), 303–314. DOI: https://doi.org/10.1007/BF02551274

[6] S. Giulini and M. Sanguineti, Approximation schemes for functional optimization problems, J. Optim. Theory Appl. 140 (2009), 33–54. DOI: https://doi.org/10.1007/s10957-008-9471-6

[7] K. Hornik, M. Stinchcombe, and H. White, Multilayer feedforward networks are universal approximatiors, Neural networks 2 (1989), 359–366. DOI: https://doi.org/10.1016/0893-6080(89)90020-8

[8] K. Hornik, Approximation capabilities of multilayer feedforward networks, Neural networks 4 (1991), no. 3, 251–257. DOI: https://doi.org/10.1016/0893-6080(91)90009-T

[9] B. Li, S. Tang, and H. Yu, Better approximations of high dimensional smooth functions by deep neural networks with rectified power units, Commun. in Comp. Phys. 27 (2020), 379–411. DOI: https://doi.org/10.4208/cicp.OA-2019-0168

[10] J. Ludwig, C. Molitor-Braun, and S. Pusti, Spectral synthesis in L2(G), Colloquium Mathematicum 138 (2015), no. 1, 89–103. DOI: https://doi.org/10.4064/cm138-1-6

[11] J. Oc´ariz, Universal approximation results on artificial neural networks, Master thesis, Univ. Rovira i Virgili, 2020.

[12] I. Ohn and Y. Kim, Smooth function approximation by deep neural networks with general activation functions, Entropy 21 (2019), no. 7, 627–411. DOI: https://doi.org/10.3390/e21070627

[13] J. Park and I. W. Sandberg, Universal approximation using radial-basis-function networks, Neural computation 3 (1991), no. 2, 246–257. DOI: https://doi.org/10.1162/neco.1991.3.2.246

[14] P. P. Petrushev, Approximation by ridge function and neural networks, SIAM Journal on Mathematical Analysis 30 (1999), 155–189. DOI: https://doi.org/10.1137/S0036141097322959

[15] W. Rudin, Real and complex analysis, 3rd ed., McGraw-Hill Book Company, 1987.

[16] W. Rudin, Fourier analysis on groups, John Wiley & Sons, 1990. DOI: https://doi.org/10.1002/9781118165621

[17] J. Schmidt-Hieber, Nonparametric regression using deep neural networks with ReLU activation function, The Annals of Statistics 48 (2020), no. 4, 1875–1897. DOI: https://doi.org/10.1214/19-AOS1875

[18] J. W. Siegel, Optimal approximation rates for deep ReLU neural networks on Sobolev and Besov spaces, Preprint, arXiv:2211.1440v6 (2024).

[19] T. Suzuki, Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal reate and curse of dimensionality, 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9 (2019), 785–79.

[20] T. D. H. van Nuland, Noncompact uniform universal approximation, Neural Networks 173 (2024), 106181. DOI: https://doi.org/10.1016/j.neunet.2024.106181

[21] F. Voigtlaender, The universal approximation theorem for complex-valued neural networks, Appl. and Comput. Harm. Anal. 64 (2023), 33–61. DOI: https://doi.org/10.1016/j.acha.2022.12.002

[22] D. Yarotsky, Error bounds for approximations with deep ReLU networks, Neural Networks 94 (2017), 103–114. DOI: https://doi.org/10.1016/j.neunet.2017.07.002