Fully Vested

The Case of Cat Modeling

Episode Summary

Jason and Graham return for a new season of Fully Vested. This time, we chat about generative artificial intelligence, its applications, and some of its discontents.

Episode Notes

Many of the core technologies behind Generative AI are not exactly brand new. For example, the "Attention Is All You Need" paper, which described and introduced the Transformer model (the "T" in ChatGPT), was published in 2017. Diffusion models—the backbone of image generation tools like StableDiffusion and DALL-e—were introduced in 2015 and were originally inspired by thermodynamic modeling techniques. Generative adversarial networks (GANs) were introduced in 2014.

However, Generative AI has seemingly taken the world by storm over the past couple years. In this episode, Graham and Jason discuss—in broad strokes—what Generative AI is, what's required to train and run foundation models, where the value lies, and frontier challenges.

Fact-Checking And Corrections

Before we begin...

At around 36:16 Jason said that the Pile was compiled by OpenAI or one of its research affiliates. This is not correct. The Pile was compiled by Eleuther.ai, and we couldn't find documentation suggesting that OpenAI incorporates the entirety of The Pile into its training data corpus.
At 49:07 Jason mentions "The Open Source Institute" but actually meant to mention the Open Source Initiative

Applied Machine Learning 101

Not all AI and applied machine learning models are created equally, and models can be designed to complete specific types of tasks. Broadly speaking, there are two types of applied machine learning models: Discriminative and Generative.

Discriminative AI

Definition: Discriminative AI focuses on learning the boundary between different classes of data from a given set of training data. Unlike generative models that learn to generate data, discriminative models learn to differentiate between classes and make predictions or decisions based on the input data.

Historical Background TLDR:

The development of Discriminative AI has its roots in statistical and machine learning approaches aimed at classification tasks.
Logistic regression and Support Vector Machines (SVMs) are early examples of discriminative models, which have been used for many years in various fields including computer vision and natural language processing.
Over time, with the development of deep learning, discriminative models like Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) have become highly effective for a wide range of classification tasks.

Pop Culture Example(s):

"Hotdog vs. Not a Hotdog algorithm" from HBO's Silicon Valley (S4E4)
Image recognition capabilities of something like Iron Man alter ego Tony Stark's JARVIS (2008)

**Real-World Example(s

Automatic speech recognition (ASR)
Spam and abuse detection
Facial recognition, such as Apple's Face ID and more Orwellian examples in places ranging from China to England

Further Reading:

Discriminative Model (Wikipedia)

Generative AI

Definition: Generative AI refers to a type of artificial intelligence that is capable of generating new data samples that are similar to a given set of training data. This is achieved through algorithms that learn the underlying patterns, structures, and distributions inherent in the training data, and can generate novel data points with similar properties.

Historical Background TLDR:

The origins of Generative AI can be traced back to the development of generative models, with early instances including probabilistic graphical models in the early 2000s.
However, the field truly began to gain traction with the advent of Generative Adversarial Networks (GANs) b y Ian Goodfellow and his colleagues in 2014.
Since then, various generative models like Variational Autoencoders (VAEs) and others have also gained prominence, contributing to the rapid advancement of Generative AI.

Pop Culture Example:

The AI from the movie Her (2013)

Real-World Example(s):

OpenAI's GPT family, alongside image models like StableDiffusion, and Midjourney.

Further Reading:

Deepgram's Generative AI page in the AI Glossary... co-written by Jason and GPT-4.
Large Language Model in the Deepgram AI Glossary... also co-written by Jason and GPT-4.
The Physics Principle That Inspired Modern AI Art (Anil Ananthaswamy, for Quanta Magazine)
Visualizing and Explaining Transformer Models From the Ground Up (Zian "Andy" Wang for the Deepgram blog, January 2023)
Transformer Explained hub on PapersWithCode
Transformers, Explained: Understand the Model Behind GPT-3, BERT, and T5 (Dale Markowitz on his blog, Dale on AI., May 2021)

Also Mentioned

Training Datasets

The Pile

ImageNet

Articles and Books

(Alleged) Leaked Google memo mentioned at around 48:05 Google "We Have No Moat, And Neither Does OpenAI (SemiAnalysis publishing a whitepaper, allegedly written by a Google staff member, which suggested that open source advancements pose an existential threat to both Google and OpenAI. May 2023)

Infinite Jest (Wikipedia page on the 1996 novel by the late David Foster Wallace), mentioned around 54:30

History of Faxes vs. Emails

In reference to 1:08:30 or thereabouts:

A fun fact which blew our mind: The earliest instances of facsimile transmission stretch back to the 1890s, and the core technology matured up through the 1940s. The first telephonic fax patent was filed in 1964 by Xerox Corporation.

The first email was sent in 1971.