Now Reading
AI Canon | Andreessen Horowitz

AI Canon | Andreessen Horowitz

2023-05-25 10:47:20

Supply: Midjourney

Analysis in synthetic intelligence is rising at an exponential fee. It’s tough for AI specialists to maintain up with all the things new being revealed, and even tougher for novices to know the place to start out.

So, on this publish, we’re sharing a curated checklist of sources we’ve relied on to get smarter about fashionable AI. We name it the “AI Canon” as a result of these papers, weblog posts, programs, and guides have had an outsized influence on the sphere over the previous a number of years.

We begin with a mild introduction to transformer and latent diffusion fashions, that are fueling the present AI wave. Subsequent, we go deep on technical studying sources; sensible guides to constructing with massive language fashions (LLMs); and evaluation of the AI market. Lastly, we embody a reference checklist of landmark analysis outcomes, beginning with “Consideration is All You Want” — the 2017 paper by Google that launched the world to transformer fashions and ushered within the age of generative AI.

These articles require no specialised background and may also help you rise up to hurry rapidly on a very powerful elements of the fashionable AI wave.

  • Software 2.0: Andrej Karpathy was one of many first to obviously clarify (in 2017!) why the brand new AI wave actually issues. His argument is that AI is a brand new and highly effective solution to program computer systems. As LLMs have improved quickly, this thesis has confirmed prescient, and it offers a superb psychological mannequin for a way the AI market could progress.
  • State of GPT: Additionally from Karpathy, this can be a very approachable rationalization of how ChatGPT / GPT fashions on the whole work, learn how to use them, and what instructions R&D could take.
  • What is ChatGPT doing … and why does it work?: Pc scientist and entrepreneur Stephen Wolfram offers an extended however extremely readable rationalization, from first ideas, of how fashionable AI fashions work. He follows the timeline from early neural nets to immediately’s LLMs and ChatGPT.
  • Transformers, explained: This publish by Dale Markowitz is a shorter, extra direct reply to the query “what’s an LLM, and the way does it work?” It is a nice solution to ease into the subject and develop instinct for the know-how. It was written about GPT-3 however nonetheless applies to newer fashions.
  • How Stable Diffusion works: That is the pc imaginative and prescient analogue to the final publish. Chris McCormick offers a layperson’s rationalization of how Secure Diffusion works and develops instinct round text-to-image fashions typically. For a good gentler introduction, try this comic from r/StableDiffusion.

These sources present a base understanding of basic concepts in machine studying and AI, from the fundamentals of deep studying to university-level programs from AI specialists.



  • Stanford CS229: Introduction to Machine Studying with Andrew Ng, protecting the basics of machine studying.
  • Stanford CS224N: NLP with Deep Studying with Chris Manning, protecting NLP fundamentals by means of the primary era of LLMs.

There are numerous sources — some higher than others — making an attempt to elucidate how LLMs work. Listed here are a few of our favorites, focusing on a variety of readers/viewers.



  • Stanford CS25: Transformers United, an internet seminar on Transformers.
  • Stanford CS324: Massive Language Fashions with Percy Liang, Tatsu Hashimoto, and Chris Re, protecting a variety of technical and non-technical features of LLMs.

Reference and commentary

  • Predictive learning, NIPS 2016: On this early speak, Yann LeCun makes a powerful case for unsupervised studying as a vital component of AI mannequin architectures at scale. Skip to 19:20 for the well-known cake analogy, which continues to be probably the greatest psychological fashions for contemporary AI.
  • AI for full-self driving at Tesla: One other traditional Karpathy speak, this time protecting the Tesla knowledge assortment engine. Beginning at 8:35 is among the nice all-time AI rants, explaining why long-tailed issues (on this case cease signal detection) are so arduous.
  • The scaling hypothesis: One of the vital shocking features of LLMs is that scaling — including extra knowledge and compute — simply retains rising accuracy. GPT-3 was the primary mannequin to reveal this clearly, and Gwern’s publish does an important job explaining the instinct behind it.
  • Chinchilla’s wild implications: Nominally an explainer of the essential Chinchilla paper (see under), this publish will get to the center of the large query in LLM scaling: are we operating out of knowledge? This builds on the publish above and offers a refreshed view on scaling legal guidelines.
  • A survey of large language models: Complete breakdown of present LLMs, together with growth timeline, measurement, coaching methods, coaching knowledge, {hardware}, and extra.
  • Sparks of artificial general intelligence: Early experiments with GPT-4: Early evaluation from Microsoft Analysis on the capabilities of GPT-4, the present most superior LLM, relative to human intelligence.
  • The AI revolution: How Auto-GPT unleashes a new era of automation and creativity: An introduction to Auto-GPT and AI brokers on the whole. This know-how could be very early however essential to grasp — it makes use of web entry and self-generated sub-tasks with a purpose to resolve particular, complicated issues or targets.
  • The Waluigi Effect: Nominally a proof of the “Waluigi impact” (i.e., why “alter egos” emerge in LLM habits), however fascinating principally for its deep dive on the speculation of LLM prompting.

A brand new software stack is rising with LLMs on the core. Whereas there isn’t plenty of formal schooling obtainable on this matter but, we pulled out a number of the most helpful sources we’ve discovered.


  • Build a GitHub support bot with GPT3, LangChain, and Python: One of many earliest public explanations of the fashionable LLM app stack. A few of the recommendation in right here is dated, however in some ways it kicked off widespread adoption and experimentation of recent AI apps.
  • Building LLM applications for production: Chip Huyen discusses lots of the key challenges in constructing LLM apps, learn how to tackle them, and what varieties of use circumstances take advantage of sense.
  • Prompt Engineering Guide: For anybody writing LLM prompts — together with app devs — that is essentially the most complete information, with particular examples for a handful of common fashions. For a lighter, extra conversational therapy, strive Brex’s prompt engineering guide.
  • Prompt injection: What’s the worst that can happen? Immediate injection is a probably critical safety vulnerability lurking for LLM apps, with no excellent resolution but. Simon Willison offers the definitive description of the issue on this publish. Almost all the things Simon writes on AI is excellent.
  • OpenAI cookbook: For builders, that is the definitive assortment of guides and code examples for working with the OpenAI API. It’s up to date frequently with new code examples.
  • Pinecone learning center: Many LLM apps are primarily based round a vector search paradigm. Pinecone’s studying middle — regardless of being branded vendor content material — affords a number of the most helpful instruction on learn how to construct on this sample.
  • LangChain docs: Because the default orchestration layer for LLM apps, LangChain connects to only about all different items of the stack. So their docs are an actual reference for the total stack and the way the items match collectively.


  • LLM Bootcamp: A sensible course for constructing LLM-based purposes with Charles Frye, Sergey Karayev, and Josh Tobin.
  • Hugging Face Transformers: Information to utilizing open-source LLMs within the Hugging Face transformers library.

LLM benchmarks

  • Chatbot Arena: An Elo-style rating system of common LLMs, led by a crew at UC Berkeley. Customers may also take part by evaluating fashions face to face.
  • Open LLM Leaderboard: A rating by Hugging Face, evaluating open supply LLMs throughout a set of normal benchmarks and duties.

We’ve all marveled at what generative AI can produce, however there are nonetheless plenty of questions on what all of it means. Which merchandise and corporations will survive and thrive? What occurs to artists? How ought to firms use it? How will it have an effect on actually jobs and society at massive? Listed here are some makes an attempt at answering these questions.

a16z pondering

  • Who owns the generative AI platform?: Our flagship evaluation of the place worth is accruing, and may accrue, on the infrastructure, mannequin, and software layers of generative AI.
  • Navigating the high cost of AI compute: An in depth breakdown of why generative AI fashions require so many computing sources, and the way to consider buying these sources (i.e., the fitting GPUs in the fitting amount, on the proper price) in a high-demand market.
  • Art isn’t dead, it’s just machine-generated: A have a look at how AI fashions have been capable of reshape inventive fields — usually assumed to be the final holdout towards automation — a lot sooner than fields comparable to software program growth.
  • The generative AI revolution in games: An in-depth evaluation from our Video games crew at how the flexibility to simply create extremely detailed graphics will change how sport designers, studios, and your complete market perform. This follow-up piece from our Video games crew appears particularly on the introduction of AI-generated content material vis à vis user-generated content material.
  • For B2B generative AI apps, is less more?: A prediction for a way LLMs will evolve on this planet of B2B enterprise purposes, centered round the concept that summarizing info will in the end be extra invaluable than producing textual content.
  • Financial services will embrace generative AI faster than you think: An argument that the monetary companies business is poised to make use of generative AI for personalised shopper experiences, cost-efficient operations, higher compliance, improved danger administration, and dynamic forecasting and reporting. 
  • Generative AI: The next consumer platform: A have a look at alternatives for generative AI to influence the buyer market throughout a variety of sectors from remedy to ecommerce.
  • To make a real difference in health care, AI will need to learn like we do: AI is poised to irrevocably change how we glance to forestall and deal with sickness. Nonetheless, to actually rework drug discovery to care supply, we must always spend money on creating an ecosystem of “specialist” AIs — that study like our greatest physicians and drug builders do immediately.
  • The new industrial revolution: Bio x AI: The subsequent industrial revolution in human historical past shall be biology powered by synthetic intelligence.

Different views

Many of the superb AI merchandise we see immediately are the results of no-less-amazing analysis, carried out by specialists inside massive firms and main universities. These days, we’ve additionally seen spectacular work from people and the open supply group taking common tasks into new instructions, for instance by creating automated brokers or porting fashions onto smaller {hardware} footprints. 

Right here’s a set of many of those papers and tasks, for people who actually wish to dive deep into generative AI. (For analysis papers and tasks, we’ve additionally included hyperlinks to the accompanying weblog posts or web sites, the place obtainable, which have a tendency to elucidate issues at the next degree. And we’ve included unique publication years so you’ll be able to observe foundational analysis over time.)

Massive language fashions

New fashions

Mannequin enhancements (e.g. fine-tuning, retrieval, consideration)

Picture era fashions


Different knowledge modalities

Code era

Video era

Human biology and medical knowledge

See Also

Audio era

Multi-dimensional picture era

Particular due to Jack Soslow, Jay Rughani, Marco Mascorro, Martin Casado, Rajko Radovanovic, and Vijay Pande for his or her contributions to this piece, and to your complete a16z crew for an all the time informative dialogue in regards to the newest in AI. And due to Sonal Chokshi and the crypto crew for constructing an extended sequence of canons on the agency.

* * *

The views expressed listed here are these of the person AH Capital Administration, L.L.C. (“a16z”) personnel quoted and usually are not the views of a16z or its associates. Sure info contained in right here has been obtained from third-party sources, together with from portfolio firms of funds managed by a16z. Whereas taken from sources believed to be dependable, a16z has not independently verified such info and makes no representations in regards to the enduring accuracy of the knowledge or its appropriateness for a given state of affairs. As well as, this content material could embody third-party commercials; a16z has not reviewed such commercials and doesn’t endorse any promoting content material contained therein.

This content material is supplied for informational functions solely, and shouldn’t be relied upon as authorized, enterprise, funding, or tax recommendation. It’s best to seek the advice of your personal advisers as to these issues. References to any securities or digital belongings are for illustrative functions solely, and don’t represent an funding advice or supply to offer funding advisory companies. Moreover, this content material will not be directed at nor meant to be used by any traders or potential traders, and should not below any circumstances be relied upon when making a choice to spend money on any fund managed by a16z. (An providing to spend money on an a16z fund shall be made solely by the personal placement memorandum, subscription settlement, and different related documentation of any such fund and must be learn of their entirety.) Any investments or portfolio firms talked about, referred to, or described usually are not consultant of all investments in automobiles managed by a16z, and there could be no assurance that the investments shall be worthwhile or that different investments made sooner or later may have related traits or outcomes. An inventory of investments made by funds managed by Andreessen Horowitz (excluding investments for which the issuer has not supplied permission for a16z to reveal publicly in addition to unannounced investments in publicly traded digital belongings) is accessible at

Charts and graphs supplied inside are for informational functions solely and shouldn’t be relied upon when making any funding choice. Previous efficiency will not be indicative of future outcomes. The content material speaks solely as of the date indicated. Any projections, estimates, forecasts, targets, prospects, and/or opinions expressed in these supplies are topic to alter with out discover and should differ or be opposite to opinions expressed by others. Please see for added essential info.

Source Link

What's Your Reaction?
In Love
Not Sure
View Comments (0)

Leave a Reply

Your email address will not be published.

2022 Blinking Robots.
WordPress by Doejo

Scroll To Top