Kolmogorov–Arnold representation theorem

In real analysis and approximation theory, the Kolmogorov–Arnold representation theorem (or superposition theorem) states that every multivariate continuous function ${\displaystyle f\colon [0,1]^{n}\to \mathbb {R} }$ can be represented as a superposition of continuous single-variable functions.

The works of Vladimir Arnold and Andrey Kolmogorov established that if f is a multivariate continuous function, then f can be written as a finite composition of continuous functions of a single variable and the binary operation of addition.[1] More specifically,

${\displaystyle f(\mathbf {x} )=f(x_{1},\ldots ,x_{n})=\sum _{q=0}^{2n}\Phi _{q}\!\left(\sum _{p=1}^{n}\phi _{q,p}(x_{p})\right)}$.

where ${\displaystyle \phi _{q,p}\colon [0,1]\to \mathbb {R} }$ and ${\displaystyle \Phi _{q}\colon \mathbb {R} \to \mathbb {R} }$.

There are proofs with specific constructions.[2]

It solved a more constrained form of Hilbert's thirteenth problem, so the original Hilbert's thirteenth problem is a corollary.[3][4][5] In a sense, they showed that the only true continuous multivariate function is the sum, since every other continuous function can be written using univariate continuous functions and summing.[6]: 180 

History

The Kolmogorov–Arnold representation theorem is closely related to Hilbert's 13th problem. In his Paris lecture at the International Congress of Mathematicians in 1900, David Hilbert formulated 23 problems which in his opinion were important for the further development of mathematics.[7] The 13th of these problems dealt with the solution of general equations of higher degrees. It is known that for algebraic equations of degree 4 the solution can be computed by formulae that only contain radicals and arithmetic operations. For higher orders, Galois theory shows us that the solutions of algebraic equations cannot be expressed in terms of basic algebraic operations. It follows from the so called Tschirnhaus transformation that the general algebraic equation

${\displaystyle x^{n}+a_{n-1}x^{n-1}+\cdots +a_{0}=0}$

can be translated to the form ${\displaystyle y^{n}+b_{n-4}y^{n-4}+\cdots +b_{1}y+1=0}$. The Tschirnhaus transformation is given by a formula containing only radicals and arithmetic operations and transforms. Therefore, the solution of an algebraic equation of degree ${\displaystyle n}$ can be represented as a superposition of functions of two variables if ${\displaystyle n<7}$ and as a superposition of functions of ${\displaystyle n-4}$ variables if ${\displaystyle n\geq 7}$. For ${\displaystyle n=7}$ the solution is a superposition of arithmetic operations, radicals, and the solution of the equation ${\displaystyle y^{7}+b_{3}y^{3}+b_{2}y^{2}+b_{1}y+1=0}$.

A further simplification with algebraic transformations seems to be impossible which led to Hilbert's conjecture that "A solution of the general equation of degree 7 cannot be represented as a superposition of continuous functions of two variables". This explains the relation of Hilbert's thirteenth problem to the representation of a higher-dimensional function as superposition of lower-dimensional functions. In this context, it has stimulated many studies in the theory of functions and other related problems by different authors.[8]

Variants

A variant of Kolmogorov's theorem that reduces the number of outer functions ${\displaystyle \Phi _{q}}$ is due to George Lorentz.[9] He showed in 1962 that the outer functions ${\displaystyle \Phi _{q}}$ can be replaced by a single function ${\displaystyle \Phi }$. More precisely, Lorentz proved the existence of functions ${\displaystyle \phi _{q,p}}$, ${\displaystyle q=0,1,\ldots ,2n}$, ${\displaystyle p=1,\ldots ,n,}$ such that

${\displaystyle f(\mathbf {x} )=\sum _{q=0}^{2n}\Phi \!\left(\sum _{p=1}^{n}\phi _{q,p}(x_{p})\right).}$

David Sprecher [10] replaced the inner functions ${\displaystyle \phi _{q,p}}$ by one single inner function with an appropriate shift in its argument. He proved that there exist real values ${\displaystyle \eta ,\lambda _{1},\ldots ,\lambda _{n}}$, a continuous function ${\displaystyle \Phi \colon \mathbb {R} \rightarrow \mathbb {R} }$, and a real increasing continuous function ${\displaystyle \phi \colon [0,1]\rightarrow [0,1]}$ with ${\displaystyle \phi \in \operatorname {Lip} (\ln 2/\ln(2N+2))}$, for ${\displaystyle N\geq n\geq 2}$, such that

${\displaystyle f(\mathbf {x} )=\sum _{q=0}^{2n}\Phi \!\left(\sum _{p=1}^{n}\lambda _{p}\phi (x_{p}+\eta q)+q\right).}$

Phillip A. Ostrand [11] generalized the Kolmogorov superposition theorem to compact metric spaces. For ${\displaystyle p=1,\ldots ,m}$ let ${\displaystyle X_{p}}$ be compact metric spaces of finite dimension ${\displaystyle n_{p}}$ and let ${\displaystyle n=\sum _{p=1}^{m}n_{p}}$. Then there exists continuous functions ${\displaystyle \phi _{q,p}\colon X_{p}\rightarrow [0,1],q=0,\ldots ,2n,p=1,\ldots ,m}$ and continuous functions ${\displaystyle G_{q}\colon [0,1]\rightarrow \mathbb {R} ,q=0,\ldots ,2n}$ such that any continuous function ${\displaystyle f\colon X_{1}\times \dots \times X_{m}\rightarrow \mathbb {R} }$ is representable in the form

${\displaystyle f(x_{1},\ldots ,x_{m})=\sum _{q=0}^{2n}G_{q}\!\left(\sum _{p=1}^{m}\phi _{q,p}(x_{p})\right).}$

Kolmogorov-Arnold representation theorem and its aforementioned variants also hold for discontinuous multivariate functions.[12]

Limitations

The theorem does not hold in general for complex multi-variate functions, as discussed here.[4] Furthermore, the non-smoothness of the inner functions and their "wild behavior" has limited the practical use of the representation,[13] although there is some debate on this.[14]

Applications

In the field of machine learning, there have been various attempts to use neural networks modeled on the Kolmogorov–Arnold representation.[15][16][17][18][19][20] In these works, the Kolmogorov–Arnold theorem plays a role analogous to that of the universal approximation theorem in the study of multilayer perceptrons.

Proof

Here one example is proved. This proof closely follows.[21] A proof for the case of functions depending on two variables is given, as the generalization is immediate.

Setup

• Let ${\textstyle I}$ be the unit interval ${\textstyle [0,1]}$.
• Let ${\textstyle C[I]}$ be the set of continuous functions of type ${\textstyle [0,1]\to \mathbb {R} }$. It is a function space with supremum norm (it is a Banach space).
• Let ${\textstyle f}$ be a continuous function of type ${\textstyle [0,1]^{2}\to \mathbb {R} }$, and let ${\textstyle \|f\|}$ be the supremum of it on ${\textstyle [0,1]^{2}}$.
• Let ${\textstyle t}$ be a positive irrational number. Its exact value is irrelevant.

We say that a 5-tuple ${\textstyle (\phi _{1},\dots ,\phi _{5})\in C[I]^{5}}$ is a Kolmogorov-Arnold tuple if and only if any ${\textstyle f\in C[I^{2}]}$ there exists a continuous function ${\textstyle g:\mathbb {R} \to \mathbb {R} }$, such that $${\displaystyle f(x,y)=\sum _{i=1}^{5}g(\phi _{i}(x)+t\phi _{i}(y))}$$In the notation, we have the following:

Theorem — The Kolmogorov–Arnold tuples make up an open and dense subset of ${\textstyle C[I]^{5}}$.

Proof

Fix a ${\textstyle f\in C[I^{2}]}$. We show that a certain subset ${\textstyle U_{f}\subset C[I]^{5}}$ is open and dense: There exists continuous ${\textstyle g}$ such that ${\textstyle \|g\|<{\frac {1.01}{7}}\|f\|}$, and $${\displaystyle {\Big \|}f(x,y)-\sum _{i=1}^{5}g(\phi _{i}(x)+t\phi _{i}(y)){\Big \|}<{\frac {6.01}{7}}\|f\|}$$We can assume that ${\textstyle \|f\|=1}$ with no loss of generality.

By continuity, the set of such 5-tuples is open in ${\textstyle C[I]^{5}}$. It remains to prove that they are dense.

The key idea is to divide ${\textstyle [0,1]^{2}}$ into an overlapping system of small squares, each with a unique address, and define ${\textstyle g}$ to have the appropriate value at each address.

Grid system

Let ${\textstyle \psi _{1}\in C[I]}$. For any ${\textstyle \epsilon >0}$, for all large ${\textstyle N}$, we can discretize ${\textstyle \psi _{1}}$ into a continuous function ${\textstyle \phi _{1}}$ satisfying the following properties:

• ${\textstyle \phi _{1}}$ is constant on each of the intervals ${\textstyle [0/5N,4/5N],[5/5N,9/5N],\dots ,[1-5/5N,1-1/5N]}$.

• These values are different rational numbers.
• ${\textstyle \|\psi _{1}-\phi _{1}\|<\epsilon }$.

This function ${\textstyle \phi _{1}}$ creates a grid address system on ${\textstyle [0,1]^{2}}$, divided into streets and blocks. The blocks are of form ${\textstyle [0/5N,4/5N]\times [0/5N,4/5N],[0/5N,4/5N]\times [5/5N,9/5N],\dots }$.

An example construction of ${\displaystyle \phi }$ and the corresponding grid system.

Since ${\textstyle f}$ is continuous on ${\textstyle [0,1]^{2}}$, it is uniformly continuous. Thus, we can take ${\textstyle N}$ large enough, so that ${\textstyle f}$ varies by less than ${\textstyle 1/7}$ on any block.

On each block, ${\textstyle \phi _{1}(x)+t\phi _{1}(y)}$ has a constant value. The key property is that, because ${\textstyle t}$ is irrational, and ${\textstyle \phi _{1}}$ is rational on the blocks, each block has a different value of ${\textstyle \phi _{1}(x)+t\phi _{1}(y)}$.

So, given any 5-tuple ${\textstyle (\psi _{1},\dots ,\psi _{5})}$, we construct such a 5-tuple ${\textstyle (\phi _{1},\dots ,\phi _{5})}$. These create 5 overlapping grid systems.

Enumerate the blocks as ${\textstyle R_{i,r}}$, where ${\textstyle R_{i,r}}$ is the ${\textstyle r}$ -th block of the grid system created by ${\textstyle \phi _{i}}$. The address of this block is ${\textstyle a_{i,r}:=\phi _{i}(x)+t\phi _{i}(y)}$, for any ${\textstyle (x,y)\in R_{i,r}}$. By adding a small and linearly independent irrational number (the construction is similar to that of the Hamel basis) to each of ${\textstyle (\phi _{1},\dots ,\phi _{5})}$, we can ensure that every block has a unique address.

By plotting out the entire grid system, one can see that every point in ${\textstyle [0,1]^{2}}$ is contained in 3 to 5 blocks, and 2 to 0 streets.

Construction of g

For each block ${\textstyle R_{i,r}}$, if ${\textstyle f>0}$ on all of ${\textstyle R_{i,r}}$ then define ${\textstyle g(a_{i,r})=+1/7}$; if ${\textstyle f<0}$ on all of ${\textstyle R_{i,r}}$ then define ${\textstyle g(a_{i,r})=-1/7}$. Now, linearly interpolate ${\textstyle g}$ between these defined values. It remains to show this construction has the desired properties.

For any ${\textstyle (x,y)\in I^{2}}$, we consider three cases.

If ${\textstyle f(x,y)\in [1/7,7/7]}$, then by uniform continuity, ${\textstyle f>0}$ on every block ${\textstyle R_{i,r}}$ that contains the point ${\textstyle (x,y)}$. This means that ${\textstyle g=1/7}$ on 3 to 5 of the blocks, and have an unknown value on 2 to 0 of the streets. Thus, we have $${\displaystyle \sum _{i=1}^{5}g(\phi _{i}(x)+t\phi _{i}(y))\in [1/7,5/7]}$$ giving$${\displaystyle {\Big |}f(x,y)-\sum _{i=1}^{5}g(\phi _{i}(x)+t\phi _{i}(y)){\Big |}\in [0,6/7]}$$Similarly for ${\textstyle f(x,y)\in [-7/7,-1/7]}$.

If ${\textstyle f(x,y)\in [-1/7,1/7]}$, then since ${\textstyle \|g\|\leq 1/7}$, we still have $${\displaystyle {\Big |}f(x,y)-\sum _{i=1}^{5}g(\phi _{i}(x)+t\phi _{i}(y)){\Big |}\in [0,6/7]}$$

Baire category theorem

Iterating the above construction, then applying the Baire category theorem, we find that the following kind of 5-tuples are open and dense in ${\displaystyle C[I]^{5}}$: There exists a sequence of ${\textstyle g_{1},g_{2},\dots }$ such that ${\textstyle \|g_{1}\|<{\frac {1.01}{7}}\|f\|}$, ${\textstyle \|g_{2}\|<{\frac {1.01}{7}}{\frac {6.01}{7}}\|f\|}$, etc. This allows their sum to be defined: ${\textstyle g:=\sum _{n}g_{n}}$, which is still continuous and bounded, and it satisfies $${\displaystyle f(x,y)=\sum _{i=1}^{5}g(\phi _{i}(x)+t\phi _{i}(y))}$$Since ${\textstyle C[I^{2}]}$ has a countable dense subset, we can apply the Baire category theorem again to obtain the full theorem.

Extensions

The above proof generalizes for ${\textstyle n}$-dimensions: Divide the cube ${\textstyle [0,1]^{n}}$ into ${\textstyle (2n+1)}$ interlocking grid systems, such that each point in the cube is on ${\textstyle (n+1)}$ to ${\textstyle (2n+1)}$ blocks, and ${\textstyle 0}$ to ${\textstyle n}$ streets. Now, since ${\textstyle (n+1)>n}$, the above construction works.

Indeed, this is the best possible value.

Theorem (Sternfeld, 1985 [22]) — Let ${\textstyle X}$ be a compact metric space with ${\textstyle \operatorname {dim} X\geq 2}$, and let ${\textstyle X\subset \mathbb {R} ^{m}}$ be an embedding such that every ${\textstyle f\in C[X]}$ can be represented as

$${\displaystyle f\left(x_{1},x_{2},\ldots ,x_{m}\right)=\sum _{i=1}^{m}g_{i}\left(x_{i}\right),\quad \left(x_{1},x_{2},\ldots ,x_{m}\right)\in X,\quad g_{i}\in C[\mathbb {R} ].}$$

Then ${\textstyle m\geq 2\operatorname {dim} X+1}$.

A relatively short proof is given in [23] via dimension theory.

In another direction of generality, more conditions can be imposed on the Kolmogorov–Arnold tuples.

Theorem — There exists a Kolmogorov–Arnold tuple where each function is strictly monotonically increasing.

The proof is given in.[24]

(Vituškin, 1954)[25] showed that the theorem is false if we require all functions ${\displaystyle f,g,\phi _{i}}$ to be continuously differentiable. The theorem remains true if we require all ${\displaystyle \phi _{i}}$ to be 1-Lipschitz continuous.[5]

References

1. Bar-Natan, Dror. "Dessert: Hilbert's 13th Problem, in Full Colour".
2. Braun, Jürgen; Griebel, Michael (2009). "On a constructive proof of Kolmogorov's superposition theorem". Constructive Approximation. 30 (3): 653–675. doi:10.1007/s00365-009-9054-2.
3. Khesin, Boris A.; Tabachnikov, Serge L. (2014). Arnold: Swimming Against the Tide. American Mathematical Society. p. 165. ISBN 978-1-4704-1699-7.
4. Akashi, Shigeo (2001). "Application of ϵ-entropy theory to Kolmogorov—Arnold representation theorem". Reports on Mathematical Physics. 48 (1–2): 19–26. doi:10.1016/S0034-4877(01)80060-4.
5. Morris, Sidney A. (2020-07-06). "Hilbert 13: Are there any genuine continuous multivariate real-valued functions?". Bulletin of the American Mathematical Society. 58 (1): 107–118. doi:10.1090/bull/1698. ISSN 0273-0979.
6. Diaconis, Persi; Shahshahani, Mehrdad (1984). "On nonlinear functions of linear combinations" (PDF). SIAM Journal on Scientific Computing. 5 (1): 175–191. doi:10.1137/0905013. Archived from the original (PDF) on 2017-08-08.
7. Hilbert, David (1902). "Mathematical problems". Bulletin of the American Mathematical Society. 8 (10): 461–462. doi:10.1090/S0002-9904-1902-00923-3.
8. Jürgen Braun, On Kolmogorov's Superposition Theorem and Its Applications, SVH Verlag, 2010, 192 pp.
9. Lorentz, G. G. (1962). "Metric entropy, widths, and superpositions of functions". American Mathematical Monthly. 69 (6): 469–485. doi:10.1080/00029890.1962.11989915.
10. Sprecher, David A. (1965). "On the Structure of Continuous Functions of Several Variables". Transactions of the American Mathematical Society. 115: 340–355. doi:10.2307/1994273. JSTOR 1994273.
11. Ostrand, Phillip A. (1965). "Dimension of metric spaces and Hilbert's problem 13". Bulletin of the American Mathematical Society. 71 (4): 619–622. doi:10.1090/s0002-9904-1965-11363-5.
12. Ismailov, Vugar (2008). "On the representation by linear superpositions". Journal of Approximation Theory. 151 (2): 113–125. arXiv:1501.05268. doi:10.1016/j.jat.2007.09.003.
13. Girosi, Federico; Poggio, Tomaso (1989). "Representation Properties of Networks: Kolmogorov's Theorem is Irrelevant". Neural Computation. 1 (4): 465–469. doi:10.1162/neco.1989.1.4.465.
14. Kůrková, Věra (1991). "Kolmogorov's Theorem is Relevant". Neural Computation. 3 (4): 617–622. doi:10.1162/neco.1991.3.4.617. PMID 31167327.
15. Lin, Ji-Nan; Unbehauen, Rolf (January 1993). "On the Realization of a Kolmogorov Network". Neural Computation. 5 (1): 18–20. doi:10.1162/neco.1993.5.1.18.
16. Köppen, Mario (2022). "On the Training of a Kolmogorov Network". Artificial Neural Networks — ICANN 2002. Lecture Notes in Computer Science. Vol. 2415. pp. 474–479. doi:10.1007/3-540-46084-5_77. ISBN 978-3-540-44074-1.
17. KAN: Kolmogorov-Arnold Networks. (Ziming Liu et al.)
18. Manon Bischoff (May 28, 2024). "An Alternative to Conventional Neural Networks Could Help Reveal What AI Is Doing behind the Scenes". Scientific American. Archived from the original on May 29, 2024. Retrieved May 29, 2024.
19. Ismayilova, Aysu; Ismailov, Vugar (August 2024). "On the Kolmogorov Neural Networks". Neural Networks. 176 (Article 106333). arXiv:2311.00049. doi:10.1016/j.neunet.2024.106333. PMID 38688072.
20. Steve Nadis (September 11, 2024). "Novel Architecture Makes Neural Networks More Understandable". Quanta Magazine.
21. Morris, Sidney (January 2021). "Hilbert 13: Are there any genuine continuous multivariate real-valued functions?". Bulletin of the American Mathematical Society. 58 (1): 107–118. doi:10.1090/bull/1698. ISSN 0273-0979.
22. Sternfeld, Y. (1985-03-01). "Dimension, superposition of functions and separation of points, in compact metric spaces". Israel Journal of Mathematics. 50 (1): 13–53. doi:10.1007/BF02761117. ISSN 1565-8511.
23. Levin, Michael (1990-06-01). "Dimension and superposition of continuous functions". Israel Journal of Mathematics. 70 (2): 205–218. doi:10.1007/BF02807868. ISSN 1565-8511.
24. Hedberg, Torbjörn (2006) [1971]. "Appendix 2: The Kolmogorov superposition theorem". In Shapiro, Harold S. (ed.). Topics in Approximation Theory. Lecture Notes in Mathematics. Vol. 187. Springer. pp. 267–275, (33– of PDF). doi:10.1007/BFb0058976. ISBN 978-3-540-36497-9.
25. Vituškin, A.G. (1954). "On Hilbert's Thirteenth Problem". Doklady Akad. Nauk SSSR (N.S.) (in Russian). 95 (4): 701–4.

Sources

• Andrey Kolmogorov, "On the representation of continuous functions of several variables by superpositions of continuous functions of a smaller number of variables", Proceedings of the USSR Academy of Sciences, 108 (1956), pp. 179–182; English translation: Amer. Math. Soc. Transl., "17: Twelve Papers on Algebra and Real Functions" (1961), pp. 369–373.
• Vladimir Arnold, "On functions of three variables", Proceedings of the USSR Academy of Sciences, 114 (1957), pp. 679–681; English translation: Amer. Math. Soc. Transl., "28: Sixteen Papers on Analysis" (1963), pp. 51–54. SpringerLink
• Vladimir Arnold, "On the representation of continuous functions of three variables as superpositions of continuous functions of two variables", Dokl. Akad. Nauk. SSSR 114:4 (1957), pp. 679–681 (in Russian) SpringerLink
• Andrey Kolmogorov, "On the representation of continuous functions of several variables as superpositions of continuous functions of one variable and addition", (1957); English translation: Amer. Math. Soc. Transl., "28: Sixteen Papers on Analysis" (1963), PDF

Further reading

• S. Ya. Khavinson, Best Approximation by Linear Superpositions (Approximate Nomography), AMS Translations of Mathematical Monographs (1997)