AI - the biggest 'game changer' Dr. Saty Raghavachary USC CS Dept, saty@usc.edu [talk link: bit.ly/CVRCOE_AI2021]

In this fast-paced (and fun!) talk, we will look at:

• the origins of AI ('phase 1')
• AI plods on... ('phase 2')
• a resurgence! ('phase 3')
• where/what next? ('phase 4')

Three things to note:

• what follows, is not on a strict timeline (ie not chronological)
• I have "incorporated" material from multiple sources
• opinions expressed are strictly mine :)

Fasten your seat belts... let's GO!

Part 1: a bold beginning

The quest for self-directed, intelligent beings goes back a LONG way. Ancient Romans and Greeks built robotic mechanisms, as did the Swiss [including cuckoo clocks].

Rossum's Universal Robots (RUR) - about a factory for making humanoids - is a 1921 sci-fi play by Karel Čapek.

McCulloch and Pitts, in 1943, published a paper - proposing the first-ever artificial neural network.

Alan Turing and von Neumann both believed in the possibility of intelligent machines. Their proposals (for the architecture) did differ - Turing wanted to build a 'child machine', whereas von Neumann thought that an 'automata' style machine would work.

Loosely speaking, the two dominant modes of AI proposed above (by McCulloch and Pitts, and by Turing and von Neumann), is what we are still pursuing - connectionist, symbolic. "But things in this life change very slowly, if they ever change at all" :)

In 1955, the pioneers of AI proposed a summer project, expecting it to lead to 'significant advances'...

The 'problem' turned out to be MUCH MORE complex, so here we are :)

The computational model of the mind that they 'had in mind', doesn't seem like all that there is.

One of the attendees of the Dartmouth Workshop, Arthur Samuel - in 1952, he wrote a checkers playing program that used heuristic search.

Allen Newell and Herbert Simon were also attendees of the Dartmouth AI workshop.

They went on to do pioneering work in AI, for which they won the 1975 Turing Award. In their Award lecture, they presented their PSSH: 'a physical symbol system [such as a digital computer, for example] has the necessary and sufficient means for intelligent action.'

Here is a thoughtful analysis on the PSSH.

The PSSH is the dominant paradigm that drives ALL of AI research today!!! Hmmmmmmmmmm... what if... what if...

Part 2: looking for breakthroughs

LISP (LIStProcessor), aka Lots-of Infernal Stupid Parentheses, or Lost In-a Sea-of Parentheses, and later, Prolog, established themselves as AI programming languages - they excel at symbol (list/sequence, tree...) processing, good for expressing logic.

The popularity of LISP/Prolog went hand-in-hand with the fact logic/symbol manipulation became the dominant mode of doing AI - "symbolic AI".

Symbols/rules/logic can capture compositionality, and causality.

This can provide 'generalization' (a capability that ML, RL struggle with, for ex).

BUT this comes at the expense of abstraction, simplification.

IF the messy, incompletely understood, confounding... real world is somehow expressed as neat rules, classified perfectly (Linnaeus, Mendeleev...), if all knowledge fits perfectly in ontologies..., THEN logic can rule!!

Interview an expert, 'mine' her brain power, create an AI expert, obsolete the real expert (the last part is not said out aloud).

What if 'expertise' in car repair, medical diagnosis, loan approval... is simply, IF-THEN-ELSE rules that experts use, even without explicitly/consciously using them, or even being aware they do?

What if we could codify the experts' rules, and have a machine employ them just as well as humans? That was supposed to be the 'promise' of ES.

So in the 80s, the AI community produced many expert systems, eg. MYCIN, PROSPECTOR, INTERNIST-1, CASNET, XCON, DELTA, JETA, AM, Eurisko [paper from '83]... "OMG".

A paragraph towards the end of the paper from above:

Doug Lenat, of AM and Eurisko fame, obtained massive funding from DoD/DOE..., to pursue a DECADE LONG (1984-1994) symbolic AI project - Cyc [short for en-Cyc-lopedia], the world's largest ever!!

Cyc had its own knowledge rep language, called CycL.

CycL was a first order (predicate) calculus language:

Cyc aimed to model the world's 'common sense', as CycL predicates (ie "rules"), resulting in a massive 'ontology' (graph) with ~100,000 nodes (entries).

Yours truly worked on Cyc, ~'92-'93! TO THIS DAY, almost DAILY, I think about 'AI', as a result of having been involved...

Cyc did not work out. Verdict: SYMBOLIC AI IS DEAD. But Cyc continues on... as a 'regular' ES engine, with NO common sense reasoning!

Symbolic AI was not the only game in town!

Rosenblatt had invented the Perceptron - a kind of neural 'network' - just ONE neuron, which linearly summed (with weighting) its inputs - it worked as long as input ('training') data was linearly separable. Here in an implementation:

BUT, Minsky and Papert, in their influential 'Perceptrons' book, showed that the architecture CAN'T work if the data is not linearly separable. UH OH - "AI winter" for Perceptrons!

BUT, they did show that adding an extra (middle, "hidden") layer of neurons CAN be used to learn linearly non-separable inputs. YEAY, resurgence!

'Connectionism' (neural networks) was pursued in many variations - CMAC, Neocognitron, ADALINE, MADALINE...

The neurons in these multilayered architectures were thought to be trainable, using differential equations!

'Backpropagation', popularized by Geoff Hinton and others, became an established, iterative way to SOLVE for each neuron's 'internal state', ie. "weights", ie. multiplier coeffs for use with input data, in the so-called "training" or "LEARNING" phase.

Throughout the late 80s, to late 90s, other approaches (not just connectionist) were tried.

Roger Schank used scripts and frames to represent knowledge - these are tailored ("scripted"), rote steps, pieces of data... that an AI agent would employ, to simulate intelligence.

In 1983, William Chamberlain created Racter, a 'paranoid, schizo' AI... This had been preceded by ELIZA, the first chatbot, written ~1966 by Weizenbaum. Such programs were the earliest attempts at 'NLP', natural language processing. To this day, natural language does not come naturally to AI!

In '97, IBM's Deep Blue used an alpha-beta search algorithm (which humans DON'T DO), to beat chess champion (unbeaten, EVER!) Garry Kasparov in a 6 game match - an incredibly unfair competition!

Also in the 90s, Rodney Brooks, Patty Maes and others, inspired by biological life, pursued 'model free' AI, when an AI agent (eg. a robot cockroach) would learn about the world, not via rules, not even via data, but by 'living' in it, ie via experience (by creating 'mental' models). This is called subsumption architecture (bottom-up processing). A runaway commercial success product, using this paradigm: the Roomba vaccum cleaner :)

Despite (or maybe because of only) small gains, AI quietly settled down into an academic activity for the most part. At conferences such as AAAI, and NIPS (now called NeurIPS), papers continued to be presented, that kept the field moving slowly, steadily - no big jumps...

AND THEN, a perfect storm of events, capabilities...

Part 3: a Cambrian explosion! [AI today...]

• the realization that GPUs can accelerate ML calcs (because weight multiplication is a dot product)!
• early SDC successes (eg Sebastian Thrun@Stanford - DARPA Grand Challenge, 2002)
• wild success with ImageNet prediction entries (Fei Fei Li, 2006)
• lots of cloud storage possibilities
• massive volumes of training data - YouTube, blogs, tweets, Google's image search...

Today's ML is, more correctly, 'supervised ML' - use past data's patterns, to 'supervise' a network's training [learning of weights and biases].

The big advance too is that our networks are DEEP, with MANY layers, AND are architectured (in terms of the connections design) in a multitude of ways [each for a specific use case].

As just mentioned, each task (eg language translation, image labeling, self-driving, Q&A, etc.) requires a specific NN architecture.

Here are popular architectures and design processes:

• deep(er) layers - so it's 'DL', rather than (shallower) 'ML'
• RNN, LSTM, HTM - earlier architectures for sequence learning (eg. words, music...)
• CNNs - for image related tasks
• CapsNet - for data-augmented traning (eg to recognize objects that are shown upisde down!)
• GAN - an AMAZING architecture that can learn to GENERATE [generator] data, because of a 'tough', "adversarial" teacher [discriminator]. GANs are quite popular!
• VAEs - a lot like GANs (can generate data), but with a different setup
• TRANSFORMERS - a "game changing" architecture that made LARGE language models possible!
• Vision Transformers - alt. to CNNs
• VideoGPT - a transfomer for generating high quality video
• neural ODEs - better error minimization
• autoML - architecture search [model optimization]
• architecture pruning - to make models lighter
• DeepRL, neuro-symbolic - combination of two approaches (the Holy Trinity of AI approaches: neural, symbolic, reinforcement)
• ...

So many libraries/APIs/environments exist, to make the process of model generation and training, easier:

• Keras, PyTorch
• mxnet
• scikit-learn
• CNTK
• Spark MLib
• Colab
• Teachable Machine
• https://www.perceptilabs.com/ - visual dataflow, no coding!

In addition to running on servers, ML can run on these:

• NVIDIA GPUs
• Google TPUs
• Coral USB
• Jetson Nano
• Intel Compute Stick
• Pixy2
• plastic film (!)
• more custom architectures in the future, thanks to FPGA/ASIC

So many companies operate in the ML 'space', but two that stand out are OpenAI and DeepMind.

More:

• Amazon uses deep learning for product recommendations, Alexa...
• Google: self-driving cars, TensorFlow, DeepMind
• Microsoft: ImageNet entry, CNTK, Skype Translator... Here are a bunch of their AI demos.
• Facebook: DeepFace can search 800M faces in <5 sec! Also, Facebook is planning to open source its hardware setup. Deep learning is also used in Instagram, for recognizing content in images, including text.
• IBM: Watson, AlchemyAPI, Watson Analytics
• Apple uses deep learning for Siri, iTunes, etc.
• Many others: Alibaba, Baidu, Tencent, Uber, Netflix, Visa, LinkedIn...

Speech, being so easy/natural has spurred a variety of thoughts/efforts:

• The Turing Test
• The Loebner Prize
• The Alexa Prize
• Alexa, Siri, OK Google, Cortona...
• chatbots on MANY retail sites, bank sites...

NLP has evolved from word2vec and skipgrams, into pretty sophisticated systems.

We can see that AI (ML) is commonplace, given that it can do so much:

• education: custom learning paths [eg http://squirrelai.com/], monitoring student engagement...
• entertainment: virtual humans, avatars..., pose estimation, eg for exercising...
• GANs: style transfer, generate music
• games: ML agents, eg in Unity
• manufacturing: Digital Twins using AR and ML
• brain scans to read mental images (!), handwriting

Predicting protein folding (w/o molecular dynamics), fluid flow w/o Navier-Stokes equations... 'w/o' because the ML learns from past (training, labeled) data.

• use of biased data
• autonomous weapons
• deepfakes [including audio, maps!]
• adversarial attacks
• ...

• XAI
• AI ethics

Here is the Turing Award lecture from Hinton, Bengio, LeCunn, just published.

From last week, here is an expose on how DL works.

Part 4: so... NOW WHAT?

There appear to be at least three directions in which things are headed:

• 'GPT-3'++: Switch Transformer, Wu Dao, etc.
• neuro-symbolic intergration - Gary Marcus et. al.
• 'RL is all you need' - "Reward is Enough"

The brain can be considered to be a 'CADS' [complex adaptive dynamical system], from whose inner workings, these emerge: thoughts, feelings, language, action [in short, everything!].

Brain STRUCTURE matters [it's a not a feature-less blackbox!].

We know so little about memory formation, recall, modification, etc.

The 'Big C' - CONSCIOUSNESS - remains a mystery, as well. The Self, "I"... "we" don't have a clear understanding, or even consensus, of what consciousness, awareness, sentience, cognition etc. are. From Sanskrit: Kasthwam Ko aham kutha ayatha? David Chalmers has called this the Hard Problem.

The 'connectome' effort hopes to understand how the brain works, by fulling mapping it out - all the ~80 billion neurons and ~1 trillion connections!

Along the way: fruit fly (Drosophila) brain, worm's (C. Elegans') full connectome [and this].

A 'Hollywood' version of this exists, as well - Baby X.

Neuromorphic computing holds a LOT of potential - use of silicon (and other materials) based neurons, analogous to the brain. Numenta has a software version of this - a tiny slice of simulated neocortex [cortical columns].

There are 'organoids' [mini brains] we can create in the lab (this raises HUGE ethics questions!); they might help figure out how our brains work.

There is a belief that machine intelligence, on account of raw computing power (regardless of underlying architecture) will match, then exceed, human intelligence - that we'd pass through the moment of 'Singularity'. My comment: "no comment!".

"ASI" - Artificial Super Intelligence - a SkyNet-like, "God-like" super intelligent entity is said to result, past singularity.

With ALL the advances in AI to date, what is STILL missing is now termed 'AGI' [to differentiate it from "mere" AI]: Artificial General Intelligence.

I do believe, strongly, in the possibility, and potential, of AGI. I also believe that what we have now, won't get us there. Melanie Mitchell, from the SFI, has similar thoughts.

What might help (achieve AGI):

• embodiment - in fact, '4E'!!
• VR "sim" will help jump-start things
• but, VR != RL - DIGITAL COMPUTATION OF A PROCESS IS NOT EQUIVALENT TO THE PROCESS!!
• this notion of intelligence: 'considered response' ('CR')
• analog (eg. neuromorphic) embodiments [for direct, physical, interactive, experiencing]!!