Jürgen Schmidhuber

Jürgen Schmidhuber
Schmidhuber speaking at the AI for GOOD Global Summit in 2017
Born17 January 1963[1]
Munich,[1] West Germany
Alma materTechnical University of Munich
Known forLong short-term memory, Gödel machine, artificial curiosity, meta-learning
Scientific career
FieldsArtificial intelligence
InstitutionsDalle Molle Institute for Artificial Intelligence Research
Websitepeople.idsia.ch/~juergen

Jürgen Schmidhuber (born 17 January 1963)[1] is a German computer scientist noted for his work in the field of artificial intelligence, specifically artificial neural networks. He is a scientific director of the Dalle Molle Institute for Artificial Intelligence Research in Switzerland.[2] He is also director of the Artificial Intelligence Initiative and professor of the Computer Science program in the Computer, Electrical, and Mathematical Sciences and Engineering (CEMSE) division at the King Abdullah University of Science and Technology (KAUST) in Saudi Arabia.[3][4]

He is best known for his foundational and highly-cited[5] work on long short-term memory (LSTM), a type of neural network architecture which was the dominant technique for various natural language processing tasks in research and commercial applications in the 2010s. He also introduced principles of dynamic neural networks, meta-learning, generative adversarial networks[6][7][8] and linear transformers,[9][10][8] all of which are widespread in modern AI.

Career

Schmidhuber completed his undergraduate (1987) and PhD (1991) studies at the Technical University of Munich in Munich, Germany.[1] His PhD advisors were Wilfried Brauer and Klaus Schulten.[11] He taught there from 2004 until 2009. From 2009,[12] until 2021, he was a professor of artificial intelligence at the Università della Svizzera Italiana in Lugano, Switzerland.[1]

He has served as the director of Dalle Molle Institute for Artificial Intelligence Research (IDSIA), a Swiss AI lab, since 1995.[1]

In 2014, Schmidhuber formed a company, Nnaisense, to work on commercial applications of artificial intelligence in fields such as finance, heavy industry and self-driving cars. Sepp Hochreiter, Jaan Tallinn, and Marcus Hutter are advisers to the company.[2] Sales were under US$11 million in 2016; however, Schmidhuber states that the current emphasis is on research and not revenue. Nnaisense raised its first round of capital funding in January 2017. Schmidhuber's overall goal is to create an all-purpose AI by training a single AI in sequence on a variety of narrow tasks.[13]

Research

In the 1980s, backpropagation did not work well for deep learning with long credit assignment paths in artificial neural networks. To overcome this problem, Schmidhuber (1991) proposed a hierarchy of recurrent neural networks (RNNs) pre-trained one level at a time by self-supervised learning.[14] It uses predictive coding to learn internal representations at multiple self-organizing time scales. This can substantially facilitate downstream deep learning. The RNN hierarchy can be collapsed into a single RNN, by distilling a higher level chunker network into a lower level automatizer network.[14][15] In 1993, a chunker solved a deep learning task whose depth exceeded 1000.[16]

In 1991, Schmidhuber published adversarial neural networks that contest with each other in the form of a zero-sum game, where one network's gain is the other network's loss.[6][17][7][8] The first network is a generative model that models a probability distribution over output patterns. The second network learns by gradient descent to predict the reactions of the environment to these patterns. This was called "artificial curiosity." In 2014, this principle was used in a generative adversarial network where the environmental reaction is 1 or 0 depending on whether the first network's output is in a given set. GANs were the state of the art in generative modeling during 2015-2020 period.

Schmidhuber supervised the 1991 diploma thesis of his student Sepp Hochreiter[18] which he considered "one of the most important documents in the history of machine learning".[15] It studied the neural history compressor,[14] and more importantly analyzed and overcame the vanishing gradient problem. This led to the long short-term memory (LSTM), a type of recurrent neural network. The name LSTM was introduced in a tech report (1995) leading to the most cited LSTM publication (1997), co-authored by Hochreiter and Schmidhuber.[19] It was not yet the standard LSTM architecture which is used in almost all current applications. The standard LSTM architecture was introduced in 2000 by Felix Gers, Schmidhuber, and Fred Cummins.[20] Today's "vanilla LSTM" using backpropagation through time was published with his student Alex Graves in 2005,[21][22] and its connectionist temporal classification (CTC) training algorithm[23] in 2006. CTC was applied to end-to-end speech recognition with LSTM. By the 2010s, the LSTM became the dominant technique for a variety of natural language processing tasks including speech recognition and machine translation, and was widely implemented in commercial technologies such as Google Neural Machine Translation,[24] have also been used in Google Voice for transcription[25] and search,[26] and Siri.[27]

In 2014, the state of the art was training “very deep neural network” with 20 to 30 layers.[28] Stacking too many layers led to a steep reduction in training accuracy,[29] known as the "degradation" problem.[30] In May 2015, Rupesh Kumar Srivastava, Klaus Greff, and Schmidhuber used LSTM principles to create the highway network, a feedforward neural network with hundreds of layers, much deeper than previous networks.[8][31][32] In Dec 2015, the residual neural network (ResNet) was published, which is a variant of the highway network.[30][33]

In 1992, Schmidhuber published fast weights programmer, an alternative to recurrent neural networks.[9] It has a slow feedforward neural network that learns by gradient descent to control the fast weights of another neural network through outer products of self-generated activation patterns, and the fast weights network itself operates over inputs.[10] This was later shown to be equivalent to the unnormalized linear Transformer.[34][10][35] Schmidhuber used the terminology "learning internal spotlights of attention" in 1993.[36]

In 2011, Schmidhuber's team at IDSIA with his postdoc Dan Ciresan also achieved dramatic speedups of convolutional neural networks (CNNs) on fast parallel computers called GPUs. An earlier CNN on GPU by Chellapilla et al. (2006) was 4 times faster than an equivalent implementation on CPU.[37] The deep CNN of Dan Ciresan et al. (2011) at IDSIA was already 60 times faster[38] and achieved the first superhuman performance in a computer vision contest in August 2011.[39] Between 15 May 2011 and 10 September 2012, these CNNs won four more image competitions[40][41] and improved the state of the art on multiple image benchmarks.[42] The approach has become central to the field of computer vision.[41] It is based on CNN designs introduced much earlier by Kunihiko Fukushima.[43][41]

Credit disputes

Schmidhuber has controversially argued that he and other researchers have been denied adequate recognition for their contribution to the field of deep learning, in favour of Geoffrey Hinton, Yoshua Bengio and Yann LeCun, who shared the 2018 Turing Award for their work in deep learning.[2][44][45] He wrote a "scathing" 2015 article arguing that Hinton, Bengio and Lecun "heavily cite each other" but "fail to credit the pioneers of the field".[45] In a statement to the New York Times, Yann LeCun wrote that "Jürgen is manically obsessed with recognition and keeps claiming credit he doesn't deserve for many, many things... It causes him to systematically stand up at the end of every talk and claim credit for what was just presented, generally not in a justified manner."[2] Schmidhuber replied that LeCun did this "without any justification, without providing a single example,"[46] and published details of numerous priority disputes with Hinton, Bengio and LeCun.[47][48]

The term "schmidhubered" has been jokingly used in the AI community to describe Schmidhuber's habit of publicly challenging the originality of other researchers' work, a practice seen by some in the AI community as a "rite of passage" for young researchers. Some suggest that Schmidhuber's significant accomplishments have been underappreciated due to his confrontational personality.[49][44]

Recognition

Schmidhuber received the Helmholtz Award of the International Neural Network Society in 2013,[50] and the Neural Networks Pioneer Award of the IEEE Computational Intelligence Society in 2016[51] for "pioneering contributions to deep learning and neural networks."[1] He is a member of the European Academy of Sciences and Arts.[52][12]

He has been referred to as the "father of modern AI" or similar,[62] the "father of Generative AI,"[63] and also the "father of deep learning."[64][55] Schmidhuber himself, however, has called Alexey Grigorevich Ivakhnenko the "father of deep learning,"[65][66] and gives credit to many even earlier AI pioneers.[15]

Views

Schmidhuber is a proponent of open source AI, and believes that they will become competitive against commercial closed-source AI.[8] He believes that AI is less threatening than nuclear weapons and does not pose a new existential threat.[58][59]

Since the 1970s, Schmidhuber wanted to create "intelligent machines that could learn and improve on their own and become smarter than him within his lifetime."[8] He differentiates between two types of AIs: tool AI, such as those for improving healthcare, and autonomous AIs that set their own goals, perform their own research, and explore the universe. He has worked on both types for decades,[8] He expects the next stage of evolution to be self-improving AIs that will succeed human civilization as the next stage in the universal increase towards ever-increasing complexity, and he expects AI to colonize the visible universe.[8]

References

  1. ^ a b c d e f g Schmidhuber, Jürgen. "Curriculum Vitae".
  2. ^ a b c d e John Markoff (27 November 2016). When A.I. Matures, It May Call Jürgen Schmidhuber ‘Dad’. The New York Times. Accessed April 2017.
  3. ^ Jürgen Schmidhuber. cemse.kaust.edu.sa. Archived from the original on 13 March 2023. Retrieved 9 May 2023.
  4. ^ "Leadership".
  5. ^ "Juergen Schmidhuber". scholar.google.com. Retrieved 20 October 2021.
  6. ^ a b Schmidhuber, Jürgen (1991). "A possibility for implementing curiosity and boredom in model-building neural controllers". Proc. SAB'1991. MIT Press/Bradford Books. pp. 222–227.
  7. ^ a b Schmidhuber, Jürgen (2020). "Generative Adversarial Networks are Special Cases of Artificial Curiosity (1990) and also Closely Related to Predictability Minimization (1991)". Neural Networks. 127: 58–66. arXiv:1906.04493. doi:10.1016/j.neunet.2020.04.008. PMID 32334341. S2CID 216056336.
  8. ^ a b c d e f g h i Jones, Hessie (23 May 2023). "Juergen Schmidhuber, Renowned 'Father Of Modern AI,' Says His Life's Work Won't Lead To Dystopia". Forbes. Retrieved 26 May 2023.
  9. ^ a b Schmidhuber, Jürgen (1 November 1992). "Learning to control fast-weight memories: an alternative to recurrent nets". Neural Computation. 4 (1): 131–139. doi:10.1162/neco.1992.4.1.131. S2CID 16683347.
  10. ^ a b c Schlag, Imanol; Irie, Kazuki; Schmidhuber, Jürgen (2021). "Linear Transformers Are Secretly Fast Weight Programmers". ICML 2021. Springer. pp. 9355–9366.
  11. ^ "Jürgen H. Schmidhuber". The Mathematics Genealogy Project. Retrieved 5 July 2022.
  12. ^ a b Dave O'Leary (3 October 2016). The Present and Future of AI and Deep Learning Featuring Professor Jürgen Schmidhuber. IT World Canada. Accessed April 2017.
  13. ^ "AI Pioneer Wants to Build the Renaissance Machine of the Future". Bloomberg.com. 16 January 2017. Retrieved 23 February 2018.
  14. ^ a b c Schmidhuber, Jürgen (1992). "Learning complex, extended sequences using the principle of history compression (based on TR FKI-148, 1991)" (PDF). Neural Computation. 4 (2): 234–242. doi:10.1162/neco.1992.4.2.234. S2CID 18271205.
  15. ^ a b c Schmidhuber, Juergen (2022). "Annotated History of Modern AI and Deep Learning". arXiv:2212.11279 [cs.NE].
  16. ^ Schmidhuber, Jürgen (1993). Habilitation Thesis (PDF).
  17. ^ Schmidhuber, Jürgen (2010). "Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990-2010)". IEEE Transactions on Autonomous Mental Development. 2 (3): 230–247. doi:10.1109/TAMD.2010.2056368. S2CID 234198.
  18. ^ S. Hochreiter., "Untersuchungen zu dynamischen neuronalen Netzen Archived 2015-03-06 at the Wayback Machine," Diploma thesis. Institut f. Informatik, Technische Univ. Munich. Advisor: J. Schmidhuber, 1991.
  19. ^ Sepp Hochreiter; Jürgen Schmidhuber (1997). "Long short-term memory". Neural Computation. 9 (8): 1735–1780. doi:10.1162/neco.1997.9.8.1735. PMID 9377276. S2CID 1915014.
  20. ^ Felix A. Gers; Jürgen Schmidhuber; Fred Cummins (2000). "Learning to Forget: Continual Prediction with LSTM". Neural Computation. 12 (10): 2451–2471. CiteSeerX 10.1.1.55.5709. doi:10.1162/089976600300015015. PMID 11032042. S2CID 11598600.
  21. ^ Graves, A.; Schmidhuber, J. (2005). "Framewise phoneme classification with bidirectional LSTM and other neural network architectures". Neural Networks. 18 (5–6): 602–610. CiteSeerX 10.1.1.331.5800. doi:10.1016/j.neunet.2005.06.042. PMID 16112549. S2CID 1856462.
  22. ^ Klaus Greff; Rupesh Kumar Srivastava; Jan Koutník; Bas R. Steunebrink; Jürgen Schmidhuber (2015). "LSTM: A Search Space Odyssey". IEEE Transactions on Neural Networks and Learning Systems. 28 (10): 2222–2232. arXiv:1503.04069. Bibcode:2015arXiv150304069G. doi:10.1109/TNNLS.2016.2582924. PMID 27411231. S2CID 3356463.
  23. ^ Graves, Alex; Fernández, Santiago; Gomez, Faustino; Schmidhuber, Juergen (2006). "Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks". In Proceedings of the International Conference on Machine Learning, ICML 2006: 369–376. CiteSeerX 10.1.1.75.6306.
  24. ^ Wu, Yonghui; Schuster, Mike; Chen, Zhifeng; Le, Quoc V.; Norouzi, Mohammad; Macherey, Wolfgang; Krikun, Maxim; Cao, Yuan; Gao, Qin; Macherey, Klaus; Klingner, Jeff; Shah, Apurva; Johnson, Melvin; Liu, Xiaobing; Kaiser, Łukasz; Gouws, Stephan; Kato, Yoshikiyo; Kudo, Taku; Kazawa, Hideto; Stevens, Keith; Kurian, George; Patil, Nishant; Wang, Wei; Young, Cliff; Smith, Jason; Riesa, Jason; Rudnick, Alex; Vinyals, Oriol; Corrado, Greg; Hughes, Macduff; Dean, Jeff (8 October 2016). "Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation". arXiv:1609.08144 [cs.CL]. Retrieved May 14, 2017
  25. ^ "The neural networks behind Google Voice transcription". 11 August 2015.
  26. ^ "Google voice search: faster and more accurate". 24 September 2015.
  27. ^ Levy, Steven (24 August 2016). "The iBrain Is Here—and It's Already Inside Your Phone". Wired. Archived from the original on 23 June 2017. Retrieved 23 June 2017.
  28. ^ Simonyan, Karen; Zisserman, Andrew (10 April 2015), Very Deep Convolutional Networks for Large-Scale Image Recognition, arXiv:1409.1556
  29. ^ He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (2016). "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification". arXiv:1502.01852 [cs.CV].
  30. ^ a b He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (10 December 2015). Deep Residual Learning for Image Recognition. arXiv:1512.03385.
  31. ^ Srivastava, Rupesh Kumar; Greff, Klaus; Schmidhuber, Jürgen (2 May 2015). "Highway Networks". arXiv:1505.00387 [cs.LG].
  32. ^ Srivastava, Rupesh K; Greff, Klaus; Schmidhuber, Juergen (2015). "Training Very Deep Networks". Advances in Neural Information Processing Systems. 28. Curran Associates, Inc.: 2377–2385.
  33. ^ He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (2016). Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE. pp. 770–778. arXiv:1512.03385. doi:10.1109/CVPR.2016.90. ISBN 978-1-4673-8851-1.
  34. ^ Katharopoulos, Angelos; Vyas, Apoorv; Pappas, Nikolaos; Fleuret, François (2020). "Transformers are RNNs: Fast autoregressive Transformers with linear attention". ICML 2020. PMLR. pp. 5156–5165.
  35. ^ Schmidhuber, Jürgen (2022). "Deep Learning: Our Miraculous Year 1990-1991". idsia.ch. Retrieved 23 July 2024.
  36. ^ Schmidhuber, Jürgen (1993). "Reducing the ratio between learning complexity and number of time-varying variables in fully recurrent nets". ICANN 1993. Springer. pp. 460–463.
  37. ^ Kumar Chellapilla; Sid Puri; Patrice Simard (2006). "High Performance Convolutional Neural Networks for Document Processing". In Lorette, Guy (ed.). Tenth International Workshop on Frontiers in Handwriting Recognition. Suvisoft.
  38. ^ Ciresan, Dan; Ueli Meier; Jonathan Masci; Luca M. Gambardella; Jurgen Schmidhuber (2011). "Flexible, High Performance Convolutional Neural Networks for Image Classification" (PDF). Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence-Volume Volume Two. 2: 1237–1242. Retrieved 17 November 2013.
  39. ^ "IJCNN 2011 Competition result table". OFFICIAL IJCNN2011 COMPETITION. 2010. Retrieved 14 January 2019.
  40. ^ Schmidhuber, Jürgen (17 March 2017). "History of computer vision contests won by deep CNNs on GPU". Retrieved 14 January 2019.
  41. ^ a b c Schmidhuber, Jürgen (2015). "Deep Learning". Scholarpedia. 10 (11): 1527–54. CiteSeerX 10.1.1.76.1541. doi:10.1162/neco.2006.18.7.1527. PMID 16764513. S2CID 2309950.
  42. ^ Ciresan, Dan; Meier, Ueli; Schmidhuber, Jürgen (June 2012). "Multi-column deep neural networks for image classification". 2012 IEEE Conference on Computer Vision and Pattern Recognition. New York, NY: Institute of Electrical and Electronics Engineers (IEEE). pp. 3642–3649. arXiv:1202.2745. CiteSeerX 10.1.1.300.3283. doi:10.1109/CVPR.2012.6248110. ISBN 978-1-4673-1226-4. OCLC 812295155. S2CID 2161592.
  43. ^ Fukushima, Neocognitron (1980). "A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position". Biological Cybernetics. 36 (4): 193–202. doi:10.1007/bf00344251. PMID 7370364. S2CID 206775608.
  44. ^ a b c Vance, Ashlee (15 May 2018). "This Man Is the Godfather the AI Community Wants to Forget". Bloomberg Business Week. Retrieved 16 January 2019.
  45. ^ a b Oltermann, Philip (18 April 2017). "Jürgen Schmidhuber on the robot future: 'They will pay as much attention to us as we do to ants'". The Guardian. Retrieved 23 February 2018.
  46. ^ Schmidhuber, Juergen (7 July 2022). "LeCun's 2022 paper on autonomous machine intelligence rehashes but does not cite essential work of 1990-2015". IDSIA, Switzerland. Archived from the original on 9 February 2023. Retrieved 3 May 2023.
  47. ^ Schmidhuber, Juergen (14 December 2023). "How 3 Turing Awardees Republished Key Methods and Ideas Whose Creators They Failed to Credit. Technical Report IDSIA-23-23". IDSIA, Switzerland. Archived from the original on 16 December 2023. Retrieved 19 December 2023.
  48. ^ Schmidhuber, Juergen (30 December 2022). "Scientific Integrity and the History of Deep Learning: The 2021 Turing Lecture, and the 2018 Turing Award. Technical Report IDSIA-77-21". IDSIA, Switzerland. Archived from the original on 7 April 2023. Retrieved 3 May 2023.
  49. ^ Fulterer, Ruth (20 February 2021). "Jürgen Schmidhuber: Tessiner Vater der künstlichen Intelligenz". Neue Zürcher Zeitung (in Swiss High German). ISSN 0376-6829. Retrieved 19 December 2023.
  50. ^ INNS Awards Recipients. International Neural Network Society. Accessed December 2016.
  51. ^ Recipients: Neural Networks Pioneer Award Archived 29 August 2021 at the Wayback Machine. Piscataway, NJ: IEEE Computational Intelligence Society. Accessed January 2019.]
  52. ^ Members. European Academy of Sciences and Arts. Accessed December 2016.
  53. ^ Heaven, Will Douglas (15 October 2020). "Artificial general intelligence: Are we close, and does it even make sense to try? Quote: Jürgen Schmidhuber—sometimes called "the father of modern AI..." MIT Technology Review. Retrieved 20 August 2021.
  54. ^ Choul-woong, Yeon (22 February 2023). "User Centric AI Creates a New Order for Users". Korea IT Times. Retrieved 26 May 2023.
  55. ^ a b Dunker, Anders (2020). "Letting loose the AI demon. Quote: But this man is no crackpot: He is the father of modern AI and deep learning – foremost in his field". Modern Times Review. Retrieved 20 August 2021.
  56. ^ Enrique Alpanes (25 April 2021). Jürgen Schmidhuber, el hombre al que Alexa y Siri llamarían ‘papá’ si él quisiera hablar con ellas. El País. Accessed August 2021.
  57. ^ Razavi, Hooman (5 May 2020). "iHuman- AI & Ethics of Cinema (2020 Hot Docs Film Festival). Quote: The documentary interviews range AI top researchers and thinkers as Jürgen Schmidhuber - Father of Modern AI..." Universal Cinema. Retrieved 20 August 2021.
  58. ^ a b Colton, Emma (7 May 2023). "'Father of AI' says tech fears misplaced: 'You cannot stop it'". Fox News. Retrieved 26 May 2023.
  59. ^ a b Taylor, Josh (7 May 2023). "Rise of artificial intelligence is inevitable but should not be feared, 'father of AI' says". The Guardian. Retrieved 26 May 2023.
  60. ^ Wong, Andrew (16 May 2018). "The 'father of A.I' urges humans not to fear the technology". CNBC. Retrieved 27 February 2019.
  61. ^ Ruth Fulterer (21 February 2021). Der unbequeme Vater der künstlichen Intelligenz lebt in der Schweiz (The inconvenient father of AI lives in Switzerland). NZZ. Accessed August 2021.
  62. ^ [8][2][53][54][55][56][57][58][59][60][61][44]
  63. ^ "Jürgen Schmidhuber：The Father of Generative AI Without Turing Award", jazzyear.com, 18 August 2024
  64. ^ Wang, Brian (14 June 2017). "Father of deep learning AI on General purpose AI and AI to conquer space in the 2050s". Next Big Future. Retrieved 27 February 2019.
  65. ^ Schmidhuber, Jurgen. "Critique of Paper by "Deep Learning Conspiracy". (Nature 521 p 436)". Retrieved 26 December 2019.
  66. ^ Ivakhnenko, A.G. (March 1970). "Heuristic self-organization in problems of engineering cybernetics". Automatica. 6 (2): 207–219. doi:10.1016/0005-1098(70)90092-0.