Pleased New Yr: GPT in 500 traces of SQL

2024-02-23 20:45:31

Translations: Russian

This 12 months, the discuss of the city was AI and the way it can do every little thing for you.

I prefer it when somebody or one thing does every little thing for me. To this finish, I made a decision to ask ChatGPT to jot down my New Yr’s publish:

“Hey ChatGPT. Are you able to implement a big language mannequin in SQL?”
“No, SQL just isn’t appropriate for implementing massive language fashions. SQL is a language for managing and querying knowledge in relational databases. Implementing a language mannequin requires complicated algorithms, massive datasets, and infrequently deep studying frameworks, that are past the capabilities of SQL.”

It simply proves that if you would like one thing completed proper, it’s a must to do it your self.

Inspired by this optimistic forecast, at the moment we’ll implement a big language mannequin in SQL.

Principle

Whereas penning this publish, I used the fantastic article GPT in 60 Lines of NumPy by Jay Mody. This text explains the interior workings of a GPT mannequin a lot better than I can hope to do. Nonetheless, just a little recap is so as.

What’s a generative massive language mannequin from a technical perspective?

A generative LLM is a operate. It takes a textual content string as enter (referred to as “immediate” in AI parlance), and returns an array of strings and numbers. Here is what the signature of this operate seems to be like:

llm(immediate: str) -> record[tuple[str, float]]

This operate is deterministic. It does loads of math below the hood, however all this math is hardwired. Should you name it repeatedly with the identical enter, it is going to at all times return the identical output.

It might come as a shock to anybody who’s been utilizing ChatGPT and comparable merchandise as a result of they can provide completely different solutions to the identical query. But, it is true. We are going to shortly see the way it works.

What are the values this operate returns?

One thing like this:


llm("I want you a contented New")

0       (' Yr', 0.967553)
1       (' Years', 0.018199688)
2       (' 12 months', 0.003573329)
3       (' York', 0.003114716)
4       (' New', 0.0009022804)
…
50252   (' carbohyd', 2.3950911e-15)
50253   (' volunte', 2.2590102e-15)
50254   ('pmwiki', 1.369229e-15)
50255   (' proport', 1.1198108e-15)
50256   (' cumbers', 7.568147e-17)

It returns an array of tuples. Every tuple consists of a phrase (or, somewhat, a string) and a quantity. The quantity is the likelihood that this phrase will proceed the immediate. The mannequin “thinks” that the phrase “I want you a contented New” shall be adopted by the character sequence ” Yr” with a likelihood of 96.7%, ” Years” of 1.8% and so forth.

The phrase “assume” above is quoted as a result of, in fact, the mannequin does not actually assume. It mechanically returns arrays of phrases and numbers based on some hardwired inside logic.

If it is that dumb and deterministic, how can it generate completely different texts?

Massive language fashions are utilized in textual content purposes (chatbots, content material mills, code assistants and many others). These purposes repeatedly name the mannequin and choose the phrase urged by it (with some extent of randomness). The following urged phrase is added to the immediate and the mannequin is named once more. This continues in a loop till sufficient phrases are generated.

The accrued sequence of phrases will appear like a textual content in a human language, full with grammar, syntax and even what seems to be intelligence and reasoning. On this side, it’s not not like a Markov chain which works on the identical precept.

The internals of a giant language mannequin are wired up in order that the following urged phrase shall be a pure continuation of the immediate, full with its grammar, semantics and sentiment. Equipping a operate with such a logic grew to become potential by a collection of scientific breakthroughs (and programming drudgery) which have resulted within the improvement of the household of algorithms referred to as GPT, or Generative Pre-trained Transformer.

What does “Generative Pre-trained Transformer” imply?

“Generative” signifies that it generates textual content (by including continuations to the immediate recursively, as we noticed earlier).

“Transformer” signifies that it makes use of a specific kind of neural community, first developed by Google and described in this paper.

“Pre-trained” is just a little bit historic. Initially, the flexibility for the mannequin to proceed textual content was regarded as only a prerequisite for a extra specialised activity: inference (discovering logical connections between phrases), classification (as an example, guessing the variety of stars in a resort score from the textual content of the evaluation), machine translation and so forth. It was thought that these two elements ought to have been educated individually, the language half being only a pre-coaching for a “actual” activity that will comply with.

As the unique GPT paper places it:

We display that enormous features on these duties might be realized by generative pre-training of a language mannequin on a various corpus of unlabeled textual content, adopted by discriminative fine-tuning on every particular activity.

It was not till later that folks realized that, with a mannequin massive sufficient, the second step was typically not mandatory. A Transformer mannequin, educated to do nothing else than generate texts, turned out to have the ability to comply with human language directions that have been contained in these texts, with no further coaching (“fine-tuning” in AI parlance) required.

With that out of the way in which, let’s give attention to the implementation.

Era

Here’s what occurs after we attempt to generate textual content from the immediate utilizing GPT2:


def generate(immediate: str) -> str:
  # Transforms a string into a listing of tokens.
  tokens = tokenize(immediate) # tokenize(immediate: str) -> record[int]

  whereas True:

    # Runs the algorithm.
    # Returns tokens' possibilities: a listing of 50257 floats, including as much as 1.
    candidates = gpt2(tokens) # gpt2(tokens: record[int]) -> record[float]

    # Selects the following token from the record of candidates
    next_token = select_next_token(candidates)
    # select_next_token(candidates: record[float]) -> int

    # Append it to the record of tokens
    tokens.append(next_token)

    # Resolve if we need to cease producing.
    # It may be token counter, timeout, stopword or one thing else.
    if should_stop_generating():
      break

  # Remodel the record of tokens right into a string
  completion = detokenize(tokens) # detokenize(tokens: record[int]) -> str
  return completion

Let’s implement all these items one after the other in SQL.

Tokenizer

Earlier than a textual content might be fed to a neural community, it must be transformed into a listing of numbers. In fact, that is barely information: that is what textual content encodings like Unicode do. Plain Unicode, nevertheless, does not actually work properly with neural networks.

Neural networks, at their core, do loads of matrix multiplications and seize no matter predictive powers they’ve within the coefficients of those matrixes. A few of these matrixes have one row per each potential worth within the “alphabet”; others have one row per “character”.

Right here, the phrases “alphabet” and “character” haven’t got the standard that means. In Unicode, the “alphabet” is 149186 characters lengthy (that is what number of completely different Unicode factors there are on the time of this writing), and a “character” might be one thing like this: ﷽ (sure, that is a single Unicode level quantity 65021, encoding a whole phrase in Arabic that’s significantly necessary for the Muslims). Word that the exact same phrase might have been written in standard Arabic letters. It signifies that the identical textual content can have many encodings.

As an illustration, let’s take the phrase “PostgreSQL”. If we have been to encode it (convert to an array of numbers) utilizing Unicode, we’d get 10 numbers that would doubtlessly be from 1 to 149186. It signifies that our neural community would want to retailer a matrix with 149186 rows in it and carry out a variety of calculations on 10 rows from this matrix. A few of these rows (equivalent to the letters of the English alphabet) could be used so much and pack loads of info; others, like poop emoji and obscure symbols from lifeless languages, would hardly be used in any respect, however nonetheless take up house.

Naturally, we need to preserve each these numbers, the “alphabet” size and the “character” rely, as little as potential. Ideally, all of the “characters” in our alphabet needs to be distributed uniformly, and we nonetheless need our encoding to be as highly effective as Unicode.

The way in which we are able to do this, intuitively, is to assign distinctive numbers to sequences of phrases that happen typically within the texts we work with. In Unicode, the identical spiritual phrase in Arabic might be encoded utilizing both a single code level, or letter by letter. Since we’re rolling our personal encoding, we are able to do the identical for the phrases and phrases which are necessary for the mannequin (i.e. present up typically in texts).

For example, we might have separate numbers for “Submit”, “greSQL” and “ing”. This fashion, the phrases “PostgreSQL” and “Posting” would each have a size of two in our illustration. And naturally, we’d nonetheless preserve separate code factors for shorter sequences and particular person bytes. Even when we come throughout gibberish or a textual content in a overseas language, it will nonetheless be encodable, albeit longer.

GPT2 makes use of a variation of the algorithm referred to as Byte pair encoding to do exactly that. Its tokenizer makes use of a dictionary of 50257 code factors (in AI parlance, “tokens”) that correspond to completely different byte sequences in UTF-8 (plus the “finish of textual content” as a separate token).

This dictionary was constructed by statistical evaluation carried out like this:

Begin with a easy encoding of 256 tokens: one token per byte.
Take a big corpus of texts (ideally the one the mannequin shall be educated on).
Encode it.
Calculate which pair of tokens is probably the most frequent. Let’s assume it is 0x20 0x74 (house adopted by the lowercase “t”).
Assign the following out there worth (257) to this pair of bytes.
Repeat the steps 3-5, now being attentive to the byte sequences. If a sequence of bytes might be encoded with a posh token, use the complicated token. If there are ambiguities (say, “abc” can, sooner or later, be encoded as “a” + “bc” or “ab” + “c”), use the one with the bottom quantity (as a result of it was added earlier and therefore is extra frequent). Do that recursively till all sequences that may collapse right into a single token will collapse right into a single token.
Carry out the collapse 50000 instances over.

The quantity 50000 was chosen kind of arbitrarily by the builders. Different fashions preserve the variety of tokens in an identical vary (from 30k to 100k).

At each iteration of this algorithm, a brand new token that could be a concatenation of two earlier ones shall be added to the dictionary. Finally, we’ll find yourself with 50256 tokens. Add a fixed-number token for “end-of-text”, and we’re completed.

The GPT2 model of BTE has one other layer of encoding: the token dictionary maps tokens to strings and never arrays of bytes. Mapping from bytes to string characters is outlined in this function. We are going to save the dictionary it produces within the desk encoder.

Let’s examine how we are able to implement the tokenizer in SQL.

The tokenizer is an integral a part of GPT2, and the token dictionary might be downloaded from OpenAI’s web site together with the remainder of the mannequin. We might want to import it into the desk tokenizer. On the backside of this publish, you’ll discover a hyperlink to the code repository. Its code will automate populating database tables wanted for the mannequin.

In a recursive CTE, we’ll break up this phrase into tokens (beginning with single bytes) and merge the most effective adjoining pairs, till there’s nothing left to merge. The merging itself occurs in a nested recursive CTE.

For the demo, I’ll use the phrase “Mississippilessly”. Every file within the resultset exhibits the most effective pair to break down discovered to this point, and likewise the progress by the question.


WITH    RECURSIVE
        bpe AS
        (
        SELECT  (n + 1)::BIGINT AS place, character, TRUE AS proceed, 1 AS step,
                NULL::INT AS token, NULL::TEXT AS mixed
        FROM    CONVERT_TO('Mississippilessly', 'UTF-8') AS bytes
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, LENGTH(bytes) - 1) AS n
        JOIN    encoder
        ON      byte = GET_BYTE(bytes, n)
        UNION ALL
        (
        WITH    RECURSIVE
                base AS
                (
                SELECT  *
                FROM    bpe
                WHERE   proceed
                ),
                bn AS
                (
                SELECT  ROW_NUMBER() OVER (ORDER BY place) AS place,
                        proceed,
                        character,
                        character || LEAD(character) OVER (ORDER BY place) AS cluster
                FROM    base
                ),
                top_rank AS
                (
                SELECT  tokenizer.*
                FROM    bn
                CROSS JOIN LATERAL
                        (
                        SELECT  *
                        FROM    tokenizer
                        WHERE   tokenizer.cluster = bn.cluster
                        LIMIT   1
                        ) tokenizer
                ORDER BY
                        token
                LIMIT   1
                ),
                breaks AS
                (
                SELECT  0::BIGINT AS place, 1 AS size
                UNION ALL
                SELECT  bn.place,
                        CASE WHEN token IS NULL THEN 1 ELSE 2 END
                FROM    breaks
                JOIN    bn
                ON      bn.place = breaks.place + size
                LEFT JOIN
                        top_rank
                USING   (cluster)
                )
        SELECT  place, character, token IS NOT NULL,
                (SELECT step + 1 FROM base LIMIT 1), token, top_rank.cluster
        FROM    breaks
        LEFT JOIN
                top_rank
        ON      1 = 1
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(character, '' ORDER BY place) AS character
                FROM    bn
                WHERE   bn.place >= breaks.place
                        AND bn.place < breaks.place + size
                ) bn
        WHERE   place > 0
        )
        )
SELECT  step, MAX(token) AS token, MAX(mixed) AS mixed, ARRAY_AGG(character ORDER BY place)
FROM    bpe
WHERE   proceed
GROUP BY
        step
ORDER BY
        step

step	token	mixed	array_agg
1	None	None	[‘M’, ‘i’, ‘s’, ‘s’, ‘i’, ‘s’, ‘s’, ‘i’, ‘p’, ‘p’, ‘i’, ‘l’, ‘e’, ‘s’, ‘s’, ‘l’, ‘y’]
2	271	is	[‘M’, ‘is’, ‘s’, ‘is’, ‘s’, ‘i’, ‘p’, ‘p’, ‘i’, ‘l’, ‘e’, ‘s’, ‘s’, ‘l’, ‘y’]
3	274	es	[‘M’, ‘is’, ‘s’, ‘is’, ‘s’, ‘i’, ‘p’, ‘p’, ‘i’, ‘l’, ‘es’, ‘s’, ‘l’, ‘y’]
4	306	ly	[‘M’, ‘is’, ‘s’, ‘is’, ‘s’, ‘i’, ‘p’, ‘p’, ‘i’, ‘l’, ‘es’, ‘s’, ‘ly’]
5	346	il	[‘M’, ‘is’, ‘s’, ‘is’, ‘s’, ‘i’, ‘p’, ‘p’, ‘il’, ‘es’, ‘s’, ‘ly’]
6	381	pp	[‘M’, ‘is’, ‘s’, ‘is’, ‘s’, ‘i’, ‘pp’, ‘il’, ‘es’, ‘s’, ‘ly’]
7	408	ess	[‘M’, ‘is’, ‘s’, ‘is’, ‘s’, ‘i’, ‘pp’, ‘il’, ‘ess’, ‘ly’]
8	747	iss	[‘M’, ‘iss’, ‘iss’, ‘i’, ‘pp’, ‘il’, ‘ess’, ‘ly’]
9	3974	ipp	[‘M’, ‘iss’, ‘iss’, ‘ipp’, ‘il’, ‘ess’, ‘ly’]
10	17140	Miss	[‘Miss’, ‘iss’, ‘ipp’, ‘il’, ‘ess’, ‘ly’]
11	30608	iless	[‘Miss’, ‘iss’, ‘ipp’, ‘iless’, ‘ly’]

On every step, the BPE algorithm finds the most effective pair of tokens to merge and merges them (you possibly can see the merged pair and its rank within the output). This process brings down the token house measurement from Unicode’s 150k to 50k, and the variety of tokens (on this specific phrase) from 17 to five. Each are nice enhancements.

When working with a number of phrases, the tokenizer first splits the textual content into separate phrases utilizing this regexp and merges the tokens inside every phrase individually. Sadly, PostgreSQL does not assist Unicode character properties in regexps, so I needed to tweak it just a little bit (in all probability killing correct Unicode assist within the course of). Here is the way it seems to be in SQL:


WITH    enter AS
        (
        SELECT  'PostgreSQL is nice' AS immediate
        ),
        clusters AS
        (
        SELECT  part_position, bpe.*
        FROM    enter
        CROSS JOIN LATERAL
                REGEXP_MATCHES(immediate, '''s|''t|''re|''ve|''m|''ll|''d| ?w+| ?d+| ?[^swd]+|s+(?!S)|s+', 'g') WITH ORDINALITY AS rm (half, part_position)
        CROSS JOIN LATERAL
                (
                WITH    RECURSIVE
                        bpe AS
                        (
                        SELECT  (n + 1)::BIGINT AS place, character, TRUE AS proceed
                        FROM    CONVERT_TO(half[1], 'UTF-8') AS bytes
                        CROSS JOIN LATERAL
                                GENERATE_SERIES(0, LENGTH(bytes) - 1) AS n
                        JOIN    encoder
                        ON      byte = GET_BYTE(bytes, n)
                        UNION ALL
                        (
                        WITH    RECURSIVE
                                base AS
                                (
                                SELECT  *
                                FROM    bpe
                                WHERE   proceed
                                ),
                                bn AS
                                (
                                SELECT  ROW_NUMBER() OVER (ORDER BY place) AS place,
                                        proceed,
                                        character,
                                        character || LEAD(character) OVER (ORDER BY place) AS cluster
                                FROM    base
                                ),
                                top_rank AS
                                (
                                SELECT  tokenizer.*
                                FROM    bn
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  *
                                        FROM    tokenizer
                                        WHERE   tokenizer.cluster = bn.cluster
                                        LIMIT   1
                                        ) tokenizer
                                ORDER BY
                                        token
                                LIMIT   1
                                ),
                                breaks AS
                                (
                                SELECT  0::BIGINT AS place, 1 AS size
                                UNION ALL
                                SELECT  bn.place,
                                        CASE WHEN token IS NULL THEN 1 ELSE 2 END
                                FROM    breaks
                                JOIN    bn
                                ON      bn.place = breaks.place + size
                                LEFT JOIN
                                        top_rank
                                USING   (cluster)
                                )
                        SELECT  place, character, token IS NOT NULL
                        FROM    breaks
                        LEFT JOIN
                                top_rank
                        ON      1 = 1
                        CROSS JOIN LATERAL
                                (
                                SELECT  STRING_AGG(character, '' ORDER BY place) AS character
                                FROM    bn
                                WHERE   bn.place >= breaks.place
                                        AND bn.place < breaks.place + size
                                ) bn
                        WHERE   place > 0
                        )
                        )
                SELECT  place, character AS cluster
                FROM    bpe
                WHERE   NOT proceed
                ) bpe
        ),
        tokens AS
        (
        SELECT  token, cluster
        FROM    clusters
        JOIN    tokenizer
        USING   (cluster)
        )
SELECT  *
FROM    tokens

token	cluster
6307	Submit
47701	greSQL
318	Ġis
1049	Ġgreat

The bizarre character Ġ is the whitespace.

This question tokenizes the immediate and converts it into an array of numbers. This fashion, the immediate is prepared for its journey by the layers of the mannequin.

Embeddings

The tokens signify elements of the human languages (about 0.75 phrases per token, on the whole), so any mannequin that’s making an attempt to succeed at textual content completion ought to in some way encode the relationships between these elements. Even in isolation, the elements of the speech have units of orthogonal properties.

Let’s take the phrase “subpoena” (which occurs to have an entire token in itself within the GPT2 tokenizer). Is it a noun? Sure, very a lot so. Is it a verb? Nicely, kind of. Is it an adjective? Not that a lot, however it may be when you squint exhausting sufficient. Is it legalese? Hell sure. And so forth.

All these properties are orthogonal, i.e. impartial of one another. A phrase generally is a legalese noun however not an adjective or a verb. In English, any mixture thereof can occur.

Issues with orthogonal properties are greatest encoded utilizing vectors. As an alternative of getting a single property (like a token quantity), we are able to have many. And it helps if we are able to wiggle them as we would like. For example, for a phrase to proceed the phrase “A court docket resolution cited by the lawyer mentions the …” we’d in all probability need one thing that is heavy on the legalese dimension and on the identical time heavy on being a noun. We do not actually care if it has a facet hustle being an adjective, a verb, or a flower.

In math, mapping narrower values into wider areas (corresponding to token IDs to vectors) is named an embedding. That is precisely what we’re doing right here.

How can we determine which properties these vectors signify? We do not. We simply present sufficient vector house for each token and hope that the mannequin throughout its coaching part will populate these dimensions with one thing significant. GPT2 makes use of 768 dimensions for its vectors. There isn’t a telling prematurely (and, truly, even within the retrospective) what property of the phrase will, say, the dimension 247 encode. Certainly it will encode one thing, however it’s not straightforward to inform what it’s.

What properties of every token can we need to embed within the vector house? Something that has any bearing on what the following token could be.

Token id? In fact. Completely different tokens imply various things.

Place of the token within the textual content? Sure, please. “Blue violet” and “violet blue” will not be the identical factor.

Relationships of tokens to one another? Positive! That is, in all probability, crucial a part of the job, and the Consideration block of the Transformer structure was the primary one to get it proper.

Tokens and positions are straightforward to embed. For example now we have the phrase “PostgreSQL is nice”, which, as we already know, maps to 4 tokens: [6307, 47701, 318, 1049].

Amongst different parameters of GPT2, there are two matrixes referred to as WTE (phrase token embedding) and WPE (phrase place embedding). Because the names recommend, the previous shops embeddings of the tokens, and the latter shops embeddings of the positions. The precise values of those embeddings have been populated (“discovered”) in the course of the coaching of GPT2. So far as we’re involved, they’re constants that reside within the database tables wte and wpe.

WTE is 50257×768 and WPE is 1024×768. The latter signifies that the utmost variety of tokens that we are able to use in a immediate to GPT2 is 1024. If we offer extra tokens within the immediate, we simply will not have the ability to pull positional embeddings for them. It is an architectural side (“hyperparameter” in AI parlance) of the mannequin that’s set at design time and can’t be modified by coaching. When individuals discuss concerning the “context window” of an LLM, they imply this quantity.

Now we have the token 6307 at place 0, 47701 at 1, 318 at 2, and 1049 at 3. For every of those tokens and positions, now we have two vectors: one from WTE and one other one from WPE. We have to add them together. The 4 ensuing vectors would be the inputs for the following a part of the algorithm: the feed-forward neural community with the eye mechanism.

For the SQL half, we’ll use pgvector, a PostgreSQL extension.

A bit of disclaimer: usually, I write code for my New Yr posts in vanilla SQL, typically with pure SQL features as helpers. It will be completely potential to do it for this publish as properly by defining vector operations on arrays, at the price of some efficiency lower (it was completed in model 1 and labored, albeit slowly). With the arrival of the AI and rising significance of vector databases, pgvector or its equal will certainly make it into the core of PostgreSQL inside two or three releases. I simply determined to experience the wave of the long run.

Here is how we do this in SQL:


WITH    embeddings AS
        (
        SELECT  place, values
        FROM    UNNEST(ARRAY[6307, 47701, 318, 1049]) WITH ORDINALITY AS tokens (token, ordinality)
        CROSS JOIN LATERAL
                (
                SELECT  ordinality - 1 AS place
                ) o
        CROSS JOIN LATERAL
                (
                SELECT  wte.values + wpe.values AS values
                FROM    wte
                CROSS JOIN
                        wpe
                WHERE   wte.token = tokens.token
                        AND wpe.place = o.place
                ) embedding
        )
SELECT  place, (values::REAL[])[0:5]
FROM    embeddings

place	values
0	[0.1035146, -0.22879261, 0.18413992, -0.29924694, 0.18642524]
1	[0.10757777, -0.0011023134, -0.0077463835, 0.03656415, -0.14654925]
2	[-0.005507436, -0.07471258, 0.11009377, -0.11708109, -0.14026159]
3	[-0.04785268, -0.0792546, 0.1628486, -0.3598496, 0.11462127]

(To maintain the output quick, this question solely exhibits the primary 5 dimensions for every vector)

Consideration

The half that basically makes the Transformer structure tick is the self-attention mechanism. It was first described within the 2017 paper “Attention is all you need” by Vasmani et al., in all probability the most well-known AI paper, whose title has since turn out to be a snowclone (a cliché for naming different papers).

Thus far, now we have a number of vectors that, hopefully, encode some syntactic and semantic properties of the phrases in our immediate. We want these properties to in some way switch to the final vector. A bit of spoiler alert: on the finish of the day, will probably be the final vector that can retailer the embedding for the continuation phrase.

In a phrase like “I appeared on the violet and noticed that it was not the standard …”, the ellipsis must be one thing you see (and this notion has to leap from “noticed”), one thing that is a property of a violet (leaping from “violet” to “it” after which to the ellipsis), and one thing that’s “uncommon” (leaping from “not” and “standard” and flipping the signal within the dimensions chargeable for the usualness). The analogy in the true world could be an individual studying a ebook in a overseas language that they type of have a fundamental command of, however do not fairly know very properly. They would want to consciously hint their method from one phrase to a different, and if they do not concentrate to the essential a part of the phrase, their understanding could be flawed.

To allow this switch of that means from one token to a different, we have to permit the vectors of all of the tokens to affect one another. If we need to populate the phrase “it” with some concrete semantics, how a lot of the semantics ought to come from the earlier vectors within the immediate, and the way a lot ought to stay from the phrase “it” itself?

To resolve this drawback, the mannequin makes use of 12 units of matrixes referred to as Q (question), Okay (key) and V (worth). Every of them has 64 columns. They’re obtained from the vector embeddings by a 768×2304 linear transformation c_attn, whose weights and biases are saved within the tables c_attn_w and c_attn_b.

The results of c_attn is a matrix with n_token rows and 2304 columns (3×12×64). It consists of 12 Q matrixes, 12 Okay matrixes and 12 V matrixes stacked horizontally, on this order.

Every set of Q, Okay and V is named a “head”. They’re used to carry out the step referred to as “multi-headed causal self-attention”, by calculating the eye operate.

Here is the formulation for the eye operate:

$A = mathrm{softmax}(dfrac{QK^T}{sqrt{d_k}} + M)V$ ,

the place softmax is the load normalization operate. It is outlined like this:

$mathrm{softmax_n}(textbf{R}) = dfrac{e^{R_n}}{sumlimits_n e^{R_n} }$

$M$ is a continuing matrix referred to as a “causal masks”. It’s outlined like this: $M = begin{bmatrix} 0 & -inf & -inf & dots & -inf 0 & 0 & -inf & dots & -inf vdots & vdots & vdots & ddots & vdots 0 & 0 & 0 & dots & -inf 0 & 0 & 0 & dots & 0 end{bmatrix}$

Softmax turns damaging infinities into zeros.

Why do we’d like masking?

The immediate in our earlier examples had 4 tokens, and the very first thing the mannequin did was calculate the 4 embeddings for these 4 tokens. Because the mannequin progresses, these vectors will endure loads of calculations, however for probably the most half, they are going to be impartial and parallel. Adjustments in a single vector won’t have an effect on the opposite vectors, as if that they had not existed. The self-attention block is the one place in the entire mannequin the place the vectors have an effect on one another.

As soon as the mannequin is completed with the mathematics, the candidates for the following token shall be determined solely from the final embedding. All the data circulation needs to be directed in the direction of this final vector and never from it. The transient values of the final embedding mustn’t have an effect on the transient values of the earlier embeddings in the course of the ahead move of the mannequin.

That is why we “masks” the latter embeddings in order that they do not affect the sooner embeddings by this specific channel. Therefore the phrase “causal” in “multi-headed causal self-attention”.

Why are the matrixes referred to as “question”, “key” and “worth”?

To be trustworthy, I am unsure it is even a great analogy. However I will nonetheless do my tackle the instinct behind it.

In machine studying, typically, calculations mustn’t contain variable-length loops or assertion branching. Every part needs to be completed by the composition of easy analytic features (additions, multiplications, powers, logarithms and trig). It permits backpropagation, which depends on applied sciences like automatic differentiation, to work effectively.

The mathematical mannequin of the key-value retailer is the expression

$displaystyle begin{cases}v, & k = q 0, & text{otherwise} end{cases}$

, however it’s not a clean, differentiable operate and it’ll not work properly with backpropagation. To make it work, we would want to show it right into a clean operate that will be shut to $v$ when $k$ is near $q$ , and shut to $0$ in any other case.

The Gaussian distribution (“bell curve”), scaled to $v$ , with the expectation of $k$ and a sufficiently small commonplace deviation would do completely for this goal:

$displaystyle frac{v}{sigmasqrt{2pi}} , mathrm{exp}left(-frac{left(q - kright)^2}{2sigma^2}right)$

, the place $sigma$ is an arbitrary parameter, defining how sharp the bell curve is.

In a vector house with many sufficient dimensions, if we take a hard and fast vector $mathbf K$ and several other vectors $mathbf Q$ that randomly and uniformly deviate from $mathbf K$ on each dimension, their dot merchandise will naturally type the bell curve. So, within the vector house, the idea of a “differentiable key-value retailer” might be modeled by the expression $(textbf Q cdot textbf K) textbf V$ , which is what we’re utilizing in our consideration operate.

Once more, this analogy is far-fetched. It is best to not pay an excessive amount of consideration (no pun supposed) to those ideas of consideration, that means circulation, hash tables and so forth. Simply consider them as an inspiration for a math trick that has been put to the take a look at and proved to work very well.

Let’s illustrate this step:


WITH    embeddings AS
        (
        SELECT  place, values
        FROM    UNNEST(ARRAY[6307, 47701, 318, 1049]) WITH ORDINALITY AS tokens (token, ordinality)
        CROSS JOIN LATERAL
                (
                SELECT  ordinality - 1 AS place
                ) o
        CROSS JOIN LATERAL
                (
                SELECT  wte.values + wpe.values AS values
                FROM    wte
                CROSS JOIN
                        wpe
                WHERE   wte.token = tokens.token
                        AND wpe.place = o.place
                ) embedding
        ),
        c_attn_w AS
        (
        SELECT  *
        FROM    c_attn_w
        WHERE   block = 0
        ),
        c_attn_b AS
        (
        SELECT  *
        FROM    c_attn_b
        WHERE   block = 0
        ),
        ln_1_g AS
        (
        SELECT  *
        FROM    ln_1_g
        WHERE   block = 0
        ),
        ln_1_b AS
        (
        SELECT  *
        FROM    ln_1_b
        WHERE   block = 0
        ),
        mha_norm AS
        (
        SELECT  place, mm.values + c_attn_b.values AS values
        FROM    (
                SELECT  place, ARRAY_AGG(INNER_PRODUCT(c_attn_w.values, layer_norm.values) ORDER BY y)::VECTOR(2304) AS values
                FROM    (
                        SELECT  place, agg.values * ln_1_g.values + ln_1_b.values AS values
                        FROM    (
                                SELECT  place, norm.values
                                FROM    embeddings
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  AVG(worth) AS imply,
                                                VAR_POP(worth) AS variance
                                        FROM    UNNEST(values::REAL[]) worth
                                        ) agg
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  ARRAY_AGG((worth - imply) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(worth, ordinality)
                                        ) norm
                                ) agg
                        CROSS JOIN
                                ln_1_b
                        CROSS JOIN
                                ln_1_g
                        ) layer_norm
                CROSS JOIN
                        c_attn_w
                GROUP BY
                        place
                ) mm
        CROSS JOIN
                c_attn_b
        ),
        head AS
        (
        SELECT  place,
                (values::REAL[])[1:64]::VECTOR(64) AS q,
                (values::REAL[])[1 + 768:64 + 768]::VECTOR(64) AS ok,
                (values::REAL[])[1 + 1536:64 + 1536]::VECTOR(64) AS v
        FROM    mha_norm
        ),
        sm_input AS
        (
        SELECT  h1.place AS x, h2.place AS y, INNER_PRODUCT(h1.q, h2.ok) / 8 + CASE WHEN h2.place > h1.place THEN -1E10 ELSE 0 END AS worth
        FROM    head h1
        CROSS JOIN
                head h2
        ),
        sm_diff AS
        (
        SELECT  x, y, worth - MAX(worth) OVER (PARTITION BY x) AS diff
        FROM    sm_input
        ),
        sm_exp AS
        (
        SELECT  x, y, CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
        FROM    sm_diff
        ),
        softmax AS
        (
        SELECT  x, y AS place, e / SUM(e) OVER (PARTITION BY x) AS worth
        FROM    sm_exp
        ),
        consideration AS
        (
        SELECT  place, (ARRAY_AGG(worth ORDER BY ordinality))[:3] AS values
        FROM    (
                SELECT  x AS place, SUM(ARRAY_FILL(softmax.worth, ARRAY[64])::VECTOR(64) * head.v) AS values
                FROM    softmax
                JOIN    head
                USING   (place)
                GROUP BY
                        x
                ) q
        CROSS JOIN LATERAL
                UNNEST(values::REAL[]) WITH ORDINALITY v (worth, ordinality)
        GROUP BY
                place
        )
SELECT  place,
        (SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((q::REAL[])[:3]) AS n) AS q,
        (SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((ok::REAL[])[:3]) AS n) AS ok,
        (SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((v::REAL[])[:3]) AS n) AS v,
        matrix,
        (SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((values::REAL[])[:3]) AS n) AS consideration
FROM    head
JOIN    consideration
USING   (place)
JOIN    (
        SELECT  x AS place, STRING_AGG(CASE WHEN worth > 0 THEN TO_CHAR(worth, '0.00') ELSE '    0' END, ' ' ORDER BY place) AS matrix
        FROM    softmax
        GROUP BY
                x
        ) softmax_grouped
USING   (place)

place	q	ok	v	matrix	consideration
0	+0.381 -0.579 +0.073 …	-1.395 +2.367 +0.332 …	-0.006 +0.192 +0.047 …	1.00 0 0 0	-0.006 +0.192 +0.047 …
1	+1.518 +0.827 -0.388 …	-2.380 +3.714 +0.659 …	-0.315 -0.062 +0.018 …	0.73 0.27 0 0	-0.089 +0.124 +0.039 …
2	+0.238 -0.226 +0.344 …	-1.952 +2.404 +1.953 …	+0.256 -0.268 +0.301 …	0.67 0.26 0.07 0	-0.069 +0.095 +0.057 …
3	+1.130 -0.011 -0.103 …	-2.855 +2.053 +2.813 …	+0.176 +0.019 -0.099 …	0.59 0.19 0.12 0.10	-0.016 +0.071 +0.058 …

Here’s what we did:

Earlier than calculating the eye operate, we normalized the vectors by making use of the linear transformation $mathbf R^prime = mathbf{RGamma_1} + mathbf{B_1}$ . The matrix $mathbf{Gamma_1}$ and the vector $mathbf{B_1}$ are referred to as “scale” and “shift”, accordingly. They’re discovered parameters of the mannequin, that are saved within the tables ln_1_g and ln_1_b
We’re solely exhibiting the primary head of the primary layer of the algorithm. After we multiplied the vectors by the discovered coefficients from c_attn_w and c_attn_b (“weight” and “bias”), we sliced the ensuing 2304-vectors, taking 64-vectors beginning on the positions 0, 768 and 1536. They correspond to the vectors Q, Okay and V for the primary head.
EXP in PostgreSQL fails on actually small numbers, that is why we shortcut to zero if the argument to EXP is lower than -745.13.
We’re solely exhibiting the primary three parts for every vector. The eye matrix we present in full.

As we are able to see, the primary worth vector obtained copied to the output as is (as it is going to do in each different layer of the algorithm). It signifies that as soon as the mannequin has been educated, the output embedding for the primary token shall be solely outlined by the worth of the primary token. Generally, in the course of the recursive inference part, the place tokens solely get added to the immediate, solely the final embedding within the output will ever change in comparison with the earlier iteration. That is what the causal masks does.

Wanting a bit ahead: the eye block is the solely place in your entire algorithm the place tokens can affect one another in the course of the ahead move. Since now we have disabled the flexibility of later tokens to affect the earlier ones on this step, all of the calculations completed on the earlier tokens might be reused between the ahead passes of the mannequin.

Keep in mind, the mannequin operates by appending tokens to the immediate. If our unique (tokenized) immediate is “Submit greSQL Ġis Ġgreat” and the following one shall be (as an example) “Submit greSQL Ġis Ġgreat Ġfor”, all the outcomes of the calculations made on the primary 4 tokens might be reused for the brand new immediate; they may by no means change, regardless of what’s appended to them.

Jay Mody’s illustrative article does not make use of this reality (and neither can we, for the sake of simplicity), however the unique GPT2 implementation does.

As soon as all of the heads are completed, we’ll find yourself with 12 matrixes, every 64 columns broad and n_tokens rows tall. To map it again to the dimension of embedding vectors (768), we simply must stack these matrixes horizontally.

The ultimate step of multi-headed consideration entails projecting the values by a discovered linear transformation of the identical dimension. Its weights and biases are saved within the tables c_proj_w and c_proj_b.

Here is what the code for a whole multi-headed consideration step within the first layer seems to be like:


WITH    embeddings AS
        (
        SELECT  place, values
        FROM    UNNEST(ARRAY[6307, 47701, 318, 1049]) WITH ORDINALITY AS tokens (token, ordinality)
        CROSS JOIN LATERAL
                (
                SELECT  ordinality - 1 AS place
                ) o
        CROSS JOIN LATERAL
                (
                SELECT  wte.values + wpe.values AS values
                FROM    wte
                CROSS JOIN
                        wpe
                WHERE   wte.token = tokens.token
                        AND wpe.place = o.place
                ) embedding
        ),
        c_proj_w AS
        (
        SELECT  *
        FROM    c_proj_w
        WHERE   block = 0
        ),
        c_proj_b AS
        (
        SELECT  *
        FROM    c_proj_b
        WHERE   block = 0
        ),
        mlp_c_fc_w AS
        (
        SELECT  *
        FROM    mlp_c_fc_w
        WHERE   block = 0
        ),
        mlp_c_fc_b AS
        (
        SELECT  *
        FROM    mlp_c_fc_b
        WHERE   block = 0
        ),
        mlp_c_proj_w AS
        (
        SELECT  *
        FROM    mlp_c_proj_w
        WHERE   block = 0
        ),
        mlp_c_proj_b AS
        (
        SELECT  *
        FROM    mlp_c_proj_b
        WHERE   block = 0
        ),
        c_attn_w AS
        (
        SELECT  *
        FROM    c_attn_w
        WHERE   block = 0
        ),
        c_attn_b AS
        (
        SELECT  *
        FROM    c_attn_b
        WHERE   block = 0
        ),
        ln_1_g AS
        (
        SELECT  *
        FROM    ln_1_g
        WHERE   block = 0
        ),
        ln_1_b AS
        (
        SELECT  *
        FROM    ln_1_b
        WHERE   block = 0
        ),
        mha_norm AS
        (
        SELECT  place, mm.values + c_attn_b.values AS values
        FROM    (
                SELECT  place, ARRAY_AGG(INNER_PRODUCT(c_attn_w.values, layer_norm.values) ORDER BY y)::VECTOR(2304) AS values
                FROM    (
                        SELECT  place, agg.values * ln_1_g.values + ln_1_b.values AS values
                        FROM    (
                                SELECT  place, norm.values
                                FROM    embeddings
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  AVG(worth) AS imply,
                                                VAR_POP(worth) AS variance
                                        FROM    UNNEST(values::REAL[]) worth
                                        ) agg
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  ARRAY_AGG((worth - imply) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(worth, ordinality)
                                        ) norm
                                ) agg
                        CROSS JOIN
                                ln_1_b
                        CROSS JOIN
                                ln_1_g
                        ) layer_norm
                CROSS JOIN
                        c_attn_w
                GROUP BY
                        place
                ) mm
        CROSS JOIN
                c_attn_b
        ),
        heads AS
        (
        SELECT  place, head,
                (values::REAL[])[(head * 64 + 1):(head * 64 + 64)]::VECTOR(64) AS q,
                (values::REAL[])[(head * 64 + 1 + 768):(head * 64 + 64 + 768)]::VECTOR(64) AS ok,
                (values::REAL[])[(head * 64 + 1 + 1536):(head * 64 + 64 + 1536)]::VECTOR(64) AS v
        FROM    mha_norm
        CROSS JOIN
                GENERATE_SERIES(0, 11) head
        ),
        sm_input AS
        (
        SELECT  head, h1.place AS x, h2.place AS y, INNER_PRODUCT(h1.q, h2.ok) / 8 + CASE WHEN h2.place > h1.place THEN -1E10 ELSE 0 END AS worth
        FROM    heads h1
        JOIN    heads h2
        USING   (head)
        ),
        sm_diff AS
        (
        SELECT  head, x, y, worth - MAX(worth) OVER (PARTITION BY head, x) AS diff
        FROM    sm_input
        ),
        sm_exp AS
        (
        SELECT  head, x, y, CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
        FROM    sm_diff
        ),
        softmax AS
        (
        SELECT  head, x, y AS place, e / SUM(e) OVER (PARTITION BY head, x) AS worth
        FROM    sm_exp
        ),
        consideration AS
        (
        SELECT  place, ARRAY_AGG(worth ORDER BY head * 64 + ordinality)::VECTOR(768) AS values
        FROM    (
                SELECT  head, x AS place, SUM(ARRAY_FILL(softmax.worth, ARRAY[64])::VECTOR(64) * heads.v) AS values
                FROM    softmax
                JOIN    heads
                USING   (head, place)
                GROUP BY
                        head, x
                ) q
        CROSS JOIN LATERAL
                UNNEST(values::REAL[]) WITH ORDINALITY v (worth, ordinality)
        GROUP BY
                place
        ),
        mha AS
        (
        SELECT  place, w.values + c_proj_b.values AS values
        FROM    (
                SELECT  consideration.place, ARRAY_AGG(INNER_PRODUCT(consideration.values, c_proj_w.values) ORDER BY c_proj_w.place)::VECTOR(768) AS values
                FROM    consideration
                CROSS JOIN
                        c_proj_w
                GROUP BY
                        consideration.place
                ) w
        CROSS JOIN
                c_proj_b
        )
SELECT  place,
        (SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((values::REAL[])[:10]) AS n) AS q
FROM    mha

place	q
0	+0.814 -1.407 +0.171 +0.008 +0.065 -0.049 -0.407 +1.178 -0.234 -0.061 …
1	+1.150 -0.430 +0.083 +0.030 +0.010 +0.015 -0.245 +3.778 -0.445 -0.004 …
2	-0.219 -0.745 -0.116 +0.032 +0.064 -0.044 +0.290 +3.187 -0.074 -0.003 …
3	-0.526 -0.757 -0.510 -0.008 +0.027 -0.017 +0.302 +2.842 +0.188 -0.028 …

Earlier than the outcomes of multi-headed consideration are handed to the following step, the unique inputs are added to them. This trick was described within the unique transformer paper. It is supposed to assist with vanishing and exploding gradients.

It is a widespread drawback throughout coaching: typically the gradients of the parameters end up too massive or too small. Altering them on the coaching iteration both has little or no impact on the loss operate (and so the mannequin converges very slowly), or, on the alternative, has such a giant impact that even a small change throws the loss operate too distant from its native minimal, negating the coaching efforts.

Feedforward

That is what the deep neural networks do. The bigger a part of the mannequin parameters is definitely used at this step.

This step is a multi-layer perceptron with three layers (768, 3072, 768), utilizing the Gaussian Error Linear Unit (GELU) as an activation operate:

$mathrm{GELU}(x) = displaystyle frac x 2 left(1 + mathrm{erf},frac x {sqrt 2}right)$
$mathrm{erf},x = displaystyle frac{2}{sqrt pi}int_0^x{e^{-t^2}},dt$

This operate has been observed to yield good ends in deep neural networks. It may be analytically approximated like this:

$mathrm{GELU}(x) displaystyle approx 0.5x left(1 + mathrm{tanh}left[0.797884left(x + 0.044715x^3right) right]right)$

The discovered linear transformation parameters for layer connections are referred to as c_fc (768 → 3072) and c_proj (3072 → 768). The values for the primary layer are first normalized utilizing the coefficients within the discovered parameter ln_2. After the feedforward step is accomplished, its enter is once more added to the output. This, too, is part of the unique transformer design.

The entire feedforward step seems to be like this:

$displaystyle mathrm{FFN}(mathbf R) = mathbf R + mathrm{c_proj}left(mathrm{GELU}left(mathrm{c_fc}left(mathrm{ln_2}left(mathbf Rright)right)right)right)$

And this is how we do that in SQL:


WITH    embeddings AS
        (
        SELECT  place, values
        FROM    UNNEST(ARRAY[6307, 47701, 318, 1049]) WITH ORDINALITY AS tokens (token, ordinality)
        CROSS JOIN LATERAL
                (
                SELECT  ordinality - 1 AS place
                ) o
        CROSS JOIN LATERAL
                (
                SELECT  wte.values + wpe.values AS values
                FROM    wte
                CROSS JOIN
                        wpe
                WHERE   wte.token = tokens.token
                        AND wpe.place = o.place
                ) embedding
        ),
        c_proj_w AS
        (
        SELECT  *
        FROM    c_proj_w
        WHERE   block = 0
        ),
        c_proj_b AS
        (
        SELECT  *
        FROM    c_proj_b
        WHERE   block = 0
        ),
        mlp_c_fc_w AS
        (
        SELECT  *
        FROM    mlp_c_fc_w
        WHERE   block = 0
        ),
        mlp_c_fc_b AS
        (
        SELECT  *
        FROM    mlp_c_fc_b
        WHERE   block = 0
        ),
        mlp_c_proj_w AS
        (
        SELECT  *
        FROM    mlp_c_proj_w
        WHERE   block = 0
        ),
        mlp_c_proj_b AS
        (
        SELECT  *
        FROM    mlp_c_proj_b
        WHERE   block = 0
        ),
        c_attn_w AS
        (
        SELECT  *
        FROM    c_attn_w
        WHERE   block = 0
        ),
        c_attn_b AS
        (
        SELECT  *
        FROM    c_attn_b
        WHERE   block = 0
        ),
        ln_1_g AS
        (
        SELECT  *
        FROM    ln_1_g
        WHERE   block = 0
        ),
        ln_1_b AS
        (
        SELECT  *
        FROM    ln_1_b
        WHERE   block = 0
        ),
        ln_2_b AS
        (
        SELECT  *
        FROM    ln_2_b
        WHERE   block = 0
        ),
        ln_2_g AS
        (
        SELECT  *
        FROM    ln_2_g
        WHERE   block = 0
        ),
        mha_norm AS
        (
        SELECT  place, mm.values + c_attn_b.values AS values
        FROM    (
                SELECT  place, ARRAY_AGG(INNER_PRODUCT(c_attn_w.values, layer_norm.values) ORDER BY y)::VECTOR(2304) AS values
                FROM    (
                        SELECT  place, agg.values * ln_1_g.values + ln_1_b.values AS values
                        FROM    (
                                SELECT  place, norm.values
                                FROM    embeddings
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  AVG(worth) AS imply,
                                                VAR_POP(worth) AS variance
                                        FROM    UNNEST(values::REAL[]) worth
                                        ) agg
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  ARRAY_AGG((worth - imply) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(worth, ordinality)
                                        ) norm
                                ) agg
                        CROSS JOIN
                                ln_1_b
                        CROSS JOIN
                                ln_1_g
                        ) layer_norm
                CROSS JOIN
                        c_attn_w
                GROUP BY
                        place
                ) mm
        CROSS JOIN
                c_attn_b
        ),
        heads AS
        (
        SELECT  place, head,
                (values::REAL[])[(head * 64 + 1):(head * 64 + 64)]::VECTOR(64) AS q,
                (values::REAL[])[(head * 64 + 1 + 768):(head * 64 + 64 + 768)]::VECTOR(64) AS ok,
                (values::REAL[])[(head * 64 + 1 + 1536):(head * 64 + 64 + 1536)]::VECTOR(64) AS v
        FROM    mha_norm
        CROSS JOIN
                GENERATE_SERIES(0, 11) head
        ),
        sm_input AS
        (
        SELECT  head, h1.place AS x, h2.place AS y, INNER_PRODUCT(h1.q, h2.ok) / 8 + CASE WHEN h2.place > h1.place THEN -1E10 ELSE 0 END AS worth
        FROM    heads h1
        JOIN    heads h2
        USING   (head)
        ),
        sm_diff AS
        (
        SELECT  head, x, y, worth - MAX(worth) OVER (PARTITION BY head, x) AS diff
        FROM    sm_input
        ),
        sm_exp AS
        (
        SELECT  head, x, y, CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
        FROM    sm_diff
        ),
        softmax AS
        (
        SELECT  head, x, y AS place, e / SUM(e) OVER (PARTITION BY head, x) AS worth
        FROM    sm_exp
        ),
        consideration AS
        (
        SELECT  place, ARRAY_AGG(worth ORDER BY head * 64 + ordinality)::VECTOR(768) AS values
        FROM    (
                SELECT  head, x AS place, SUM(ARRAY_FILL(softmax.worth, ARRAY[64])::VECTOR(64) * heads.v) AS values
                FROM    softmax
                JOIN    heads
                USING   (head, place)
                GROUP BY
                        head, x
                ) q
        CROSS JOIN LATERAL
                UNNEST(values::REAL[]) WITH ORDINALITY v (worth, ordinality)
        GROUP BY
                place
        ),
        mha AS
        (
        SELECT  place, w.values + c_proj_b.values + embeddings.values AS values
        FROM    (
                SELECT  consideration.place, ARRAY_AGG(INNER_PRODUCT(consideration.values, c_proj_w.values) ORDER BY c_proj_w.place)::VECTOR(768) AS values
                FROM    consideration
                CROSS JOIN
                        c_proj_w
                GROUP BY
                        consideration.place
                ) w
        CROSS JOIN
                c_proj_b
        JOIN    embeddings
        USING   (place)
        ),
        ffn_norm AS
        (
        SELECT  place, agg.values * ln_2_g.values + ln_2_b.values AS values
        FROM    (
                SELECT  place, norm.values
                FROM    mha
                CROSS JOIN LATERAL
                        (
                        SELECT  AVG(worth) AS imply,
                                VAR_POP(worth) AS variance
                        FROM    UNNEST(values::REAL[]) worth
                        ) agg
                CROSS JOIN LATERAL
                        (
                        SELECT  ARRAY_AGG((worth - imply) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(worth, ordinality)
                        ) norm
                ) agg
        CROSS JOIN
                ln_2_b
        CROSS JOIN
                ln_2_g
        ),
        ffn_a AS
        (
        SELECT  gelu.place, gelu.values
        FROM    (
                SELECT  place, w.values + mlp_c_fc_b.values AS values
                FROM    (
                        SELECT  ffn_norm.place, ARRAY_AGG(INNER_PRODUCT(ffn_norm.values, mlp_c_fc_w.values) ORDER BY mlp_c_fc_w.place)::VECTOR(3072) AS values
                        FROM    ffn_norm
                        CROSS JOIN
                                mlp_c_fc_w
                        GROUP BY
                                ffn_norm.place
                        ) w
                CROSS JOIN
                        mlp_c_fc_b
                ) v
        CROSS JOIN LATERAL
                (
                SELECT  place, ARRAY_AGG(0.5 * worth * (1 + TANH(0.797884560802 * (worth + 0.044715 * worth*worth*worth))) ORDER BY ordinality)::VECTOR(3072) AS values
                FROM    UNNEST(values::REAL[]) WITH ORDINALITY n (worth, ordinality)
                GROUP BY
                        place
                ) gelu
        ),
        ffn AS
        (
        SELECT  place, w.values + mlp_c_proj_b.values + mha.values AS values
        FROM    (
                SELECT  ffn_a.place, ARRAY_AGG(INNER_PRODUCT(ffn_a.values, mlp_c_proj_w.values) ORDER BY mlp_c_proj_w.place)::VECTOR(768) AS values
                FROM    ffn_a
                CROSS JOIN
                        mlp_c_proj_w
                GROUP BY
                        ffn_a.place
                ) w
        CROSS JOIN
                mlp_c_proj_b
        JOIN    mha
        USING   (place)
        )
SELECT  place,
        (SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((values::REAL[])[:10]) AS n) AS q
FROM    ffn

place	q
0	+0.309 -1.267 -0.250 -1.111 -0.226 +0.549 -0.346 +0.645 -1.603 -0.501 …
1	+0.841 -1.081 +0.227 -1.029 -1.554 +1.061 -0.070 +5.258 -1.892 -0.973 …
2	-1.256 -0.528 -0.846 -0.288 +0.166 +0.409 +0.019 +3.393 +0.085 -0.212 …
3	-1.007 -1.719 -0.725 -1.417 -0.086 -0.144 +0.605 +3.272 +1.051 -0.666 …

This output is what comes out of the primary block of GPT2.

Blocks

What we noticed within the earlier steps is repeated in layers (referred to as “blocks”). The blocks are arrange in a pipeline in order that the output of a earlier block goes straight to the following one. Every block has its personal set of discovered parameters.

In SQL, we would want to attach the blocks utilizing a recursive CTE.

As soon as the ultimate block produces the values, we have to normalize it utilizing the discovered parameter ln_f.

Here is what the mannequin in the end seems to be like:

$displaystyle mathrm{GPT}(tokens) = mathrm{ln_f}(mathbf R_{12})$

$displaystyle mathbf R_{n} = mathrm{block_n}(mathbf R_{n-1}), n > 0$

$displaystyle mathrm{block_n}(mathbf R) = mathrm{ffn_n}(mathrm{mha_n}(mathbf R))$

$displaystyle mathbf R_0 = mathrm{wte}(tokens) + mathrm{wpe}([1 ldots mathrm{dim}(tokens)])$

And this is the way it seems to be in SQL:


WITH    RECURSIVE
        preliminary AS
        (
        SELECT  ARRAY[6307, 47701, 318, 1049] AS enter
        ),
        hparams AS
        (
        SELECT  12 AS n_block
        ),
        embeddings AS
        (
        SELECT  place, values
        FROM    preliminary
        CROSS JOIN
                hparams
        CROSS JOIN LATERAL
                UNNEST(enter) WITH ORDINALITY AS tokens (token, ordinality)
        CROSS JOIN LATERAL
                (
                SELECT  ordinality - 1 AS place
                ) o
        CROSS JOIN LATERAL
                (
                SELECT  wte.values + wpe.values AS values
                FROM    wte
                CROSS JOIN
                        wpe
                WHERE   wte.token = tokens.token
                        AND wpe.place = o.place
                ) embedding
        ),
        remodel AS
        (
        SELECT  0 AS block, place, values
        FROM    embeddings
        UNION ALL
        (
        WITH    earlier AS
                (
                SELECT  *
                FROM    remodel
                )
        SELECT  block + 1 AS block, transformed_layer.*
        FROM    hparams
        CROSS JOIN LATERAL
                (
                SELECT  block
                FROM    earlier
                WHERE   block < 12
                LIMIT   1
                ) q
        CROSS JOIN LATERAL
                (
                WITH    ln_2_b AS
                        (
                        SELECT  *
                        FROM    ln_2_b
                        WHERE   block = q.block
                        ),
                        ln_2_g AS
                        (
                        SELECT  *
                        FROM    ln_2_g
                        WHERE   block = q.block
                        ),
                        c_proj_w AS
                        (
                        SELECT  *
                        FROM    c_proj_w
                        WHERE   block = q.block
                        ),
                        c_proj_b AS
                        (
                        SELECT  *
                        FROM    c_proj_b
                        WHERE   block = q.block
                        ),
                        mlp_c_fc_w AS
                        (
                        SELECT  *
                        FROM    mlp_c_fc_w
                        WHERE   block = q.block
                        ),
                        mlp_c_fc_b AS
                        (
                        SELECT  *
                        FROM    mlp_c_fc_b
                        WHERE   block = q.block
                        ),
                        mlp_c_proj_w AS
                        (
                        SELECT  *
                        FROM    mlp_c_proj_w
                        WHERE   block = q.block
                        ),
                        mlp_c_proj_b AS
                        (
                        SELECT  *
                        FROM    mlp_c_proj_b
                        WHERE   block = q.block
                        ),
                        c_attn_w AS
                        (
                        SELECT  *
                        FROM    c_attn_w
                        WHERE   block = q.block
                        ),
                        c_attn_b AS
                        (
                        SELECT  *
                        FROM    c_attn_b
                        WHERE   block = q.block
                        ),
                        ln_1_g AS
                        (
                        SELECT  *
                        FROM    ln_1_g
                        WHERE   block = q.block
                        ),
                        ln_1_b AS
                        (
                        SELECT  *
                        FROM    ln_1_b
                        WHERE   block = q.block
                        ),
                        mha_norm AS
                        (
                        SELECT  place, mm.values + c_attn_b.values AS values
                        FROM    (
                                SELECT  place, ARRAY_AGG(INNER_PRODUCT(c_attn_w.values, layer_norm.values) ORDER BY y)::VECTOR(2304) AS values
                                FROM    (
                                        SELECT  place, agg.values * ln_1_g.values + ln_1_b.values AS values
                                        FROM    (
                                                SELECT  place, norm.values
                                                FROM    earlier
                                                CROSS JOIN LATERAL
                                                        (
                                                        SELECT  AVG(worth) AS imply,
                                                                VAR_POP(worth) AS variance
                                                        FROM    UNNEST(values::REAL[]) worth
                                                        ) agg
                                                CROSS JOIN LATERAL
                                                        (
                                                        SELECT  ARRAY_AGG((worth - imply) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(worth, ordinality)
                                                        ) norm
                                                ) agg
                                        CROSS JOIN
                                                ln_1_b
                                        CROSS JOIN
                                                ln_1_g
                                        ) layer_norm
                                CROSS JOIN
                                        c_attn_w
                                GROUP BY
                                        place
                                ) mm
                        CROSS JOIN
                                c_attn_b
                        ),
                        heads AS
                        (
                        SELECT  place, head,
                                (values::REAL[])[(head * 64 + 1):(head * 64 + 64)]::VECTOR(64) AS q,
                                (values::REAL[])[(head * 64 + 1 + 768):(head * 64 + 64 + 768)]::VECTOR(64) AS ok,
                                (values::REAL[])[(head * 64 + 1 + 1536):(head * 64 + 64 + 1536)]::VECTOR(64) AS v
                        FROM    mha_norm
                        CROSS JOIN
                                GENERATE_SERIES(0, 11) head
                        ),
                        sm_input AS
                        (
                        SELECT  head, h1.place AS x, h2.place AS y, INNER_PRODUCT(h1.q, h2.ok) / 8 + CASE WHEN h2.place > h1.place THEN -1E10 ELSE 0 END AS worth
                        FROM    heads h1
                        JOIN    heads h2
                        USING   (head)
                        ),
                        sm_diff AS
                        (
                        SELECT  head, x, y, worth - MAX(worth) OVER (PARTITION BY head, x) AS diff
                        FROM    sm_input
                        ),
                        sm_exp AS
                        (
                        SELECT  head, x, y, CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
                        FROM    sm_diff
                        ),
                        softmax AS
                        (
                        SELECT  head, x, y AS place, e / SUM(e) OVER (PARTITION BY head, x) AS worth
                        FROM    sm_exp
                        ),
                        consideration AS
                        (
                        SELECT  place, ARRAY_AGG(worth ORDER BY head * 64 + ordinality)::VECTOR(768) AS values
                        FROM    (
                                SELECT  head, x AS place, SUM(ARRAY_FILL(softmax.worth, ARRAY[64])::VECTOR(64) * heads.v) AS values
                                FROM    softmax
                                JOIN    heads
                                USING   (head, place)
                                GROUP BY
                                        head, x
                                ) q
                        CROSS JOIN LATERAL
                                UNNEST(values::REAL[]) WITH ORDINALITY v (worth, ordinality)
                        GROUP BY
                                place
                        ),
                        mha AS
                        (
                        SELECT  place, w.values + c_proj_b.values + earlier.values AS values
                        FROM    (
                                SELECT  consideration.place, ARRAY_AGG(INNER_PRODUCT(consideration.values, c_proj_w.values) ORDER BY c_proj_w.place)::VECTOR(768) AS values
                                FROM    consideration
                                CROSS JOIN
                                        c_proj_w
                                GROUP BY
                                        consideration.place
                                ) w
                        CROSS JOIN
                                c_proj_b
                        JOIN    earlier
                        USING   (place)
                        ),
                        ffn_norm AS
                        (
                        SELECT  place, agg.values * ln_2_g.values + ln_2_b.values AS values
                        FROM    (
                                SELECT  place, norm.values
                                FROM    mha
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  AVG(worth) AS imply,
                                                VAR_POP(worth) AS variance
                                        FROM    UNNEST(values::REAL[]) worth
                                        ) agg
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  ARRAY_AGG((worth - imply) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(worth, ordinality)
                                        ) norm
                                ) agg
                        CROSS JOIN
                                ln_2_b
                        CROSS JOIN
                                ln_2_g
                        ),
                        ffn_a AS
                        (
                        SELECT  gelu.place, gelu.values
                        FROM    (
                                SELECT  place, w.values + mlp_c_fc_b.values AS values
                                FROM    (
                                        SELECT  ffn_norm.place, ARRAY_AGG(INNER_PRODUCT(ffn_norm.values, mlp_c_fc_w.values) ORDER BY mlp_c_fc_w.place)::VECTOR(3072) AS values
                                        FROM    ffn_norm
                                        CROSS JOIN
                                                mlp_c_fc_w
                                        GROUP BY
                                                ffn_norm.place
                                        ) w
                                CROSS JOIN
                                        mlp_c_fc_b
                                ) v
                        CROSS JOIN LATERAL
                                (
                                SELECT  place, ARRAY_AGG(0.5 * worth * (1 + TANH(0.797884560802 * (worth + 0.044715 * worth*worth*worth))) ORDER BY ordinality)::VECTOR(3072) AS values
                                FROM    UNNEST(values::REAL[]) WITH ORDINALITY n (worth, ordinality)
                                GROUP BY
                                        place
                                ) gelu
                        ),
                        ffn AS
                        (
                        SELECT  place, w.values + mlp_c_proj_b.values + mha.values AS values
                        FROM    (
                                SELECT  ffn_a.place, ARRAY_AGG(INNER_PRODUCT(ffn_a.values, mlp_c_proj_w.values) ORDER BY mlp_c_proj_w.place)::VECTOR(768) AS values
                                FROM    ffn_a
                                CROSS JOIN
                                        mlp_c_proj_w
                                GROUP BY
                                        ffn_a.place
                                ) w
                        CROSS JOIN
                                mlp_c_proj_b
                        JOIN    mha
                        USING   (place)
                        )
                SELECT  *
                FROM    ffn
                ) transformed_layer
        )
        ),
        block_output AS
        (
        SELECT  *
        FROM    hparams
        JOIN    remodel
        ON      remodel.block = n_block
        ),
        ln_f AS
        (
        SELECT  place, norm.values * ln_f_g.values + ln_f_b.values AS values
        FROM    block_output
        CROSS JOIN LATERAL
                (
                SELECT  AVG(worth) AS imply,
                        VAR_POP(worth) AS variance
                FROM    UNNEST(values::REAL[]) AS n(worth)
                ) agg
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG((worth - imply) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n (worth, ordinality)
                ) norm
        CROSS JOIN
                ln_f_b
        CROSS JOIN
                ln_f_g
        )
SELECT  place,
        (SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((values::REAL[])[:10]) AS n) AS q
FROM    ln_f

place	q
0	-0.153 -0.126 -0.368 +0.028 -0.013 -0.198 +0.661 +0.056 -0.228 -0.001 …
1	-0.157 -0.314 +0.291 -0.386 -0.273 -0.054 +3.397 +0.440 -0.137 -0.243 …
2	-0.912 -0.220 -0.886 -0.661 +0.491 -0.050 +0.693 +1.128 +0.031 -0.577 …
3	-0.098 -0.323 -1.479 -0.736 +0.235 -0.608 +1.774 +0.566 -0.057 -0.211 …

That is the output of the mannequin.

The fourth vector is the precise embedding of the following token predicted by the mannequin. We simply must map it again to the tokens.

Tokens

Now we have an embedding (a 768-vector) which, based on the mannequin, captures the semantics and the grammar of the probably continuation of the immediate. Now we have to map it again to the token.

One of many first steps the mannequin makes is mapping the tokens to their embeddings. It’s completed by the 50257×768 matrix wpe. We might want to use the identical matrix to map the embedding again to the token.

The issue is that the precise reverse mapping just isn’t potential: the embedding won’t (seemingly) be equal to any of the rows within the matrix. So we might want to discover the “closest” token to the embedding.

Because the dimensions of embeddings seize (as we hope) some semantic and grammatical elements of the token, we’d like them to match as intently as potential. One strategy to consolidate the closeness of every dimension could be to only calculate the dot product of the 2 embeddings. The upper the dot product, the nearer the token is to the prediction.

To do that, we’ll multiply the embedding by the matrix wte. The consequence shall be a single-column matrix, 50257 rows tall. Every worth on this consequence would be the dot product of the anticipated embedding and the token embedding. The upper this quantity, the extra seemingly it’s for the token to proceed the immediate.

To select the following token, we might want to convert the similarities to possibilities. To do that, we’ll use our good good friend softmax (the identical operate that we used to normalize consideration weights).

Why use softmax for possibilities?

Softmax has the great property of satisfying Luce’s choice axiom. It signifies that the relative possibilities of two choices do not rely on the presence or likelihood of different choices. If A is twice as possible as B, then the presence or absence of different choices won’t change this ratio (though it in fact can change absolutely the values).

The vector of dot merchandise (“logit” in AI parlance) incorporates arbitrary scores that do not have an intrinsic scale. If A has a bigger rating than B, we all know that it is extra seemingly, however that is about it. We will tweak the inputs to softmax as we please, so long as they preserve their order (i.e. bigger scores keep bigger).

One widespread method to try this is to normalize the scores by subtracting the best worth from the set from them (in order that the largest rating turns into 0 and the remaining turn out to be damaging numbers). Then we take some mounted quantity (as an example 5 or ten) high scores. Lastly, we multiply every rating by a relentless earlier than feeding it to softmax.

The variety of high scores that we take is often referred to as $top_n$ and the multiplication fixed (or, somewhat, its reverse) is named “temperature” ( $T$ ). The upper the temperature, the extra smoothed out the possibilities, and the larger the possibility that the following picked token won’t be simply the primary one.

The formulation for tokens’ possibilities is $displaystyle p_n = mathrm{softmax_nleft(frac{mathbf{scores}}{T}right)}$ , the place $mathbf{scores}$ is the set of $top_n$ scores.

Why is it referred to as “temperature”?

The softmax operate has one other title: Boltzmann distribution. It is extensively utilized in physics. Amongst different issues, it serves as a base for the barometric formula, which tells how density or air varies with altitude.

Intuitively, scorching air rises. It spreads additional away from the Earth. When air is scorching, it is extra seemingly for an air molecule to bounce off its neighbors and leap at an in any other case inconceivable peak. In comparison with colder temperatures, air density will increase at increased altitudes and drops at sea stage.

See how air behaves at completely different temperatures:

Courtesy of Dominic Ford, Bouncing Balls and the Boltzmann Distribution

By analogy, a big “temperature” will increase the likelihood of second-choice tokens being chosen (on the expense of the first-choice tokens, in fact). The inference turns into much less predictable and extra “artistic”.

Let’s put this all into SQL. The immediate was “PostgreSQL is nice”. Listed below are the highest 5 tokens that, based on the mannequin, are probably to proceed this phrase, and their possibilities at completely different temperatures:


WITH    RECURSIVE
        preliminary AS
        (
        SELECT  ARRAY[6307, 47701, 318, 1049] AS enter
        ),
        hparams AS
        (
        SELECT  12 AS n_block,
                5 AS top_n,
                ARRAY_LENGTH(enter, 1) AS n_seq
        FROM    preliminary
        ),
        embeddings AS
        (
        SELECT  place, values
        FROM    preliminary
        CROSS JOIN
                hparams
        CROSS JOIN LATERAL
                UNNEST(enter) WITH ORDINALITY AS tokens (token, ordinality)
        CROSS JOIN LATERAL
                (
                SELECT  ordinality - 1 AS place
                ) o
        CROSS JOIN LATERAL
                (
                SELECT  wte.values + wpe.values AS values
                FROM    wte
                CROSS JOIN
                        wpe
                WHERE   wte.token = tokens.token
                        AND wpe.place = o.place
                ) embedding
        ),
        remodel AS
        (
        SELECT  0 AS block, place, values
        FROM    embeddings
        UNION ALL
        (
        WITH    earlier AS
                (
                SELECT  *
                FROM    remodel
                )
        SELECT  block + 1 AS block, transformed_layer.*
        FROM    hparams
        CROSS JOIN LATERAL
                (
                SELECT  block
                FROM    earlier
                WHERE   block < 12
                LIMIT   1
                ) q
        CROSS JOIN LATERAL
                (
                WITH    ln_2_b AS
                        (
                        SELECT  *
                        FROM    ln_2_b
                        WHERE   block = q.block
                        ),
                        ln_2_g AS
                        (
                        SELECT  *
                        FROM    ln_2_g
                        WHERE   block = q.block
                        ),
                        c_proj_w AS
                        (
                        SELECT  *
                        FROM    c_proj_w
                        WHERE   block = q.block
                        ),
                        c_proj_b AS
                        (
                        SELECT  *
                        FROM    c_proj_b
                        WHERE   block = q.block
                        ),
                        mlp_c_fc_w AS
                        (
                        SELECT  *
                        FROM    mlp_c_fc_w
                        WHERE   block = q.block
                        ),
                        mlp_c_fc_b AS
                        (
                        SELECT  *
                        FROM    mlp_c_fc_b
                        WHERE   block = q.block
                        ),
                        mlp_c_proj_w AS
                        (
                        SELECT  *
                        FROM    mlp_c_proj_w
                        WHERE   block = q.block
                        ),
                        mlp_c_proj_b AS
                        (
                        SELECT  *
                        FROM    mlp_c_proj_b
                        WHERE   block = q.block
                        ),
                        c_attn_w AS
                        (
                        SELECT  *
                        FROM    c_attn_w
                        WHERE   block = q.block
                        ),
                        c_attn_b AS
                        (
                        SELECT  *
                        FROM    c_attn_b
                        WHERE   block = q.block
                        ),
                        ln_1_g AS
                        (
                        SELECT  *
                        FROM    ln_1_g
                        WHERE   block = q.block
                        ),
                        ln_1_b AS
                        (
                        SELECT  *
                        FROM    ln_1_b
                        WHERE   block = q.block
                        ),
                        mha_norm AS
                        (
                        SELECT  place, mm.values + c_attn_b.values AS values
                        FROM    (
                                SELECT  place, ARRAY_AGG(INNER_PRODUCT(c_attn_w.values, layer_norm.values) ORDER BY y)::VECTOR(2304) AS values
                                FROM    (
                                        SELECT  place, agg.values * ln_1_g.values + ln_1_b.values AS values
                                        FROM    (
                                                SELECT  place, norm.values
                                                FROM    earlier
                                                CROSS JOIN LATERAL
                                                        (
                                                        SELECT  AVG(worth) AS imply,
                                                                VAR_POP(worth) AS variance
                                                        FROM    UNNEST(values::REAL[]) worth
                                                        ) agg
                                                CROSS JOIN LATERAL
                                                        (
                                                        SELECT  ARRAY_AGG((worth - imply) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(worth, ordinality)
                                                        ) norm
                                                ) agg
                                        CROSS JOIN
                                                ln_1_b
                                        CROSS JOIN
                                                ln_1_g
                                        ) layer_norm
                                CROSS JOIN
                                        c_attn_w
                                GROUP BY
                                        place
                                ) mm
                        CROSS JOIN
                                c_attn_b
                        ),
                        heads AS
                        (
                        SELECT  place, head,
                                (values::REAL[])[(head * 64 + 1):(head * 64 + 64)]::VECTOR(64) AS q,
                                (values::REAL[])[(head * 64 + 1 + 768):(head * 64 + 64 + 768)]::VECTOR(64) AS ok,
                                (values::REAL[])[(head * 64 + 1 + 1536):(head * 64 + 64 + 1536)]::VECTOR(64) AS v
                        FROM    mha_norm
                        CROSS JOIN
                                GENERATE_SERIES(0, 11) head
                        ),
                        sm_input AS
                        (
                        SELECT  head, h1.place AS x, h2.place AS y, INNER_PRODUCT(h1.q, h2.ok) / 8 + CASE WHEN h2.place > h1.place THEN -1E10 ELSE 0 END AS worth
                        FROM    heads h1
                        JOIN    heads h2
                        USING   (head)
                        ),
                        sm_diff AS
                        (
                        SELECT  head, x, y, worth - MAX(worth) OVER (PARTITION BY head, x) AS diff
                        FROM    sm_input
                        ),
                        sm_exp AS
                        (
                        SELECT  head, x, y, CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
                        FROM    sm_diff
                        ),
                        softmax AS
                        (
                        SELECT  head, x, y AS place, e / SUM(e) OVER (PARTITION BY head, x) AS worth
                        FROM    sm_exp
                        ),
                        consideration AS
                        (
                        SELECT  place, ARRAY_AGG(worth ORDER BY head * 64 + ordinality)::VECTOR(768) AS values
                        FROM    (
                                SELECT  head, x AS place, SUM(ARRAY_FILL(softmax.worth, ARRAY[64])::VECTOR(64) * heads.v) AS values
                                FROM    softmax
                                JOIN    heads
                                USING   (head, place)
                                GROUP BY
                                        head, x
                                ) q
                        CROSS JOIN LATERAL
                                UNNEST(values::REAL[]) WITH ORDINALITY v (worth, ordinality)
                        GROUP BY
                                place
                        ),
                        mha AS
                        (
                        SELECT  place, w.values + c_proj_b.values + earlier.values AS values
                        FROM    (
                                SELECT  consideration.place, ARRAY_AGG(INNER_PRODUCT(consideration.values, c_proj_w.values) ORDER BY c_proj_w.place)::VECTOR(768) AS values
                                FROM    consideration
                                CROSS JOIN
                                        c_proj_w
                                GROUP BY
                                        consideration.place
                                ) w
                        CROSS JOIN
                                c_proj_b
                        JOIN    earlier
                        USING   (place)
                        ),
                        ffn_norm AS
                        (
                        SELECT  place, agg.values * ln_2_g.values + ln_2_b.values AS values
                        FROM    (
                                SELECT  place, norm.values
                                FROM    mha
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  AVG(worth) AS imply,
                                                VAR_POP(worth) AS variance
                                        FROM    UNNEST(values::REAL[]) worth
                                        ) agg
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  ARRAY_AGG((worth - imply) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(worth, ordinality)
                                        ) norm
                                ) agg
                        CROSS JOIN
                                ln_2_b
                        CROSS JOIN
                                ln_2_g
                        ),
                        ffn_a AS
                        (
                        SELECT  gelu.place, gelu.values
                        FROM    (
                                SELECT  place, w.values + mlp_c_fc_b.values AS values
                                FROM    (
                                        SELECT  ffn_norm.place, ARRAY_AGG(INNER_PRODUCT(ffn_norm.values, mlp_c_fc_w.values) ORDER BY mlp_c_fc_w.place)::VECTOR(3072) AS values
                                        FROM    ffn_norm
                                        CROSS JOIN
                                                mlp_c_fc_w
                                        GROUP BY
                                                ffn_norm.place
                                        ) w
                                CROSS JOIN
                                        mlp_c_fc_b
                                ) v
                        CROSS JOIN LATERAL
                                (
                                SELECT  place, ARRAY_AGG(0.5 * worth * (1 + TANH(0.797884560802 * (worth + 0.044715 * worth*worth*worth))) ORDER BY ordinality)::VECTOR(3072) AS values
                                FROM    UNNEST(values::REAL[]) WITH ORDINALITY n (worth, ordinality)
                                GROUP BY
                                        place
                                ) gelu
                        ),
                        ffn AS
                        (
                        SELECT  place, w.values + mlp_c_proj_b.values + mha.values AS values
                        FROM    (
                                SELECT  ffn_a.place, ARRAY_AGG(INNER_PRODUCT(ffn_a.values, mlp_c_proj_w.values) ORDER BY mlp_c_proj_w.place)::VECTOR(768) AS values
                                FROM    ffn_a
                                CROSS JOIN
                                        mlp_c_proj_w
                                GROUP BY
                                        ffn_a.place
                                ) w
                        CROSS JOIN
                                mlp_c_proj_b
                        JOIN    mha
                        USING   (place)
                        )
                SELECT  *
                FROM    ffn
                ) transformed_layer
        )
        ),
        block_output AS
        (
        SELECT  *
        FROM    hparams
        JOIN    remodel
        ON      remodel.block = n_block
        ),
        ln_f AS
        (
        SELECT  place, norm.values * ln_f_g.values + ln_f_b.values AS values
        FROM    block_output
        CROSS JOIN LATERAL
                (
                SELECT  AVG(worth) AS imply,
                        VAR_POP(worth) AS variance
                FROM    UNNEST(values::REAL[]) AS n(worth)
                ) agg
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG((worth - imply) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n (worth, ordinality)
                ) norm
        CROSS JOIN
                ln_f_b
        CROSS JOIN
                ln_f_g
        ),
        logits AS
        (
        SELECT  logits.*
        FROM    hparams
        CROSS JOIN LATERAL
                (
                SELECT  token, INNER_PRODUCT(ln_f.values, wte.values) AS worth
                FROM    ln_f
                CROSS JOIN
                        wte
                WHERE   ln_f.place = n_seq - 1
                ORDER BY
                        worth DESC
                LIMIT   (top_n)
                ) logits
        ),
        temperatures (temperature) AS
        (
        VALUES
        (0.5),
        (1),
        (2)
        ),
        tokens AS
        (
        SELECT  token, worth, softmax, temperature
        FROM    temperatures
        CROSS JOIN LATERAL
                (
                SELECT  *, (e / SUM(e) OVER ()) AS softmax
                FROM    (
                        SELECT  *,
                                (worth - MAX(worth) OVER ()) / temperature AS diff
                        FROM    logits
                        ) exp_x
                CROSS JOIN LATERAL
                        (
                        SELECT  CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
                        ) exp
                ) q
        )
SELECT  token,
        cluster,
        TO_CHAR(t1.worth, 'S00.000') AS rating,
        TO_CHAR(t1.softmax, '0.00') AS "temperature = 0.5",
        TO_CHAR(t2.softmax, '0.00') AS "temperature = 1",
        TO_CHAR(t3.softmax, '0.00') AS "temperature = 2"
FROM    (
        SELECT  *
        FROM    tokens
        WHERE   temperature = 0.5
        ) t1
JOIN    (
        SELECT  *
        FROM    tokens
        WHERE   temperature = 1
        ) t2
USING   (token)
JOIN    (
        SELECT  *
        FROM    tokens
        WHERE   temperature = 2
        ) t3
USING   (token)
JOIN    tokenizer
USING   (token)

token	cluster	rating	temperature = 0.5	temperature = 1	temperature = 2
329	Ġfor	-85.435	0.74	0.48	0.33
11	,	-86.232	0.15	0.22	0.22
13	.	-86.734	0.05	0.13	0.17
379	Ġat	-86.785	0.05	0.12	0.17
284	Ġto	-87.628	0.01	0.05	0.11

Inference

Lastly, we’re able to do some actual inference: run the mannequin, choose a token based on its likelihood, add it to the immediate and repeat till sufficient tokens are generated.

The LLM itself, as we noticed earlier than, is deterministic: it is only a collection of matrix multiplications and different math operations on predefined constants. So long as the immediate and the hyperparameters like temperature and top_n are the identical, the output may even be the identical.

The one non-deterministic course of is token choice. There may be randomness concerned in it (to a variable diploma). That is why GPT-based chatbots can provide completely different solutions to the identical immediate.

We are going to use the phrase “Pleased New Yr! I want” because the immediate and make the mannequin generate 10 new tokens for this immediate. The temperature shall be set to 2, and top_n shall be set to five.

The question runs for two:44 minutes on my machine. Here is its output:


SELECT SETSEED(0.20231231);

WITH    RECURSIVE
        enter AS
        (
        SELECT  'Pleased New Yr! I want you' AS immediate,
                10 AS threshold,
                2 AS temperature,
                1 AS top_n
        ),
        clusters AS
        (
        SELECT  part_position, bpe.*
        FROM    enter
        CROSS JOIN LATERAL
                REGEXP_MATCHES(immediate, '''s|''t|''re|''ve|''m|''ll|''d| ?w+| ?d+| ?[^swd]+|s+(?!S)|s+', 'g') WITH ORDINALITY AS rm (half, part_position)
        CROSS JOIN LATERAL
                (
                WITH    RECURSIVE
                        bpe AS
                        (
                        SELECT  (n + 1)::BIGINT AS place, character, TRUE AS proceed
                        FROM    CONVERT_TO(half[1], 'UTF-8') AS bytes
                        CROSS JOIN LATERAL
                                GENERATE_SERIES(0, LENGTH(bytes) - 1) AS n
                        JOIN    encoder
                        ON      byte = GET_BYTE(bytes, n)
                        UNION ALL
                        (
                        WITH    RECURSIVE
                                base AS
                                (
                                SELECT  *
                                FROM    bpe
                                WHERE   proceed
                                ),
                                bn AS
                                (
                                SELECT  ROW_NUMBER() OVER (ORDER BY place) AS place,
                                        proceed,
                                        character,
                                        character || LEAD(character) OVER (ORDER BY place) AS cluster
                                FROM    base
                                ),
                                top_rank AS
                                (
                                SELECT  tokenizer.*
                                FROM    bn
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  *
                                        FROM    tokenizer
                                        WHERE   tokenizer.cluster = bn.cluster
                                        LIMIT   1
                                        ) tokenizer
                                ORDER BY
                                        token
                                LIMIT   1
                                ),
                                breaks AS
                                (
                                SELECT  0::BIGINT AS place, 1 AS size
                                UNION ALL
                                SELECT  bn.place,
                                        CASE WHEN token IS NULL THEN 1 ELSE 2 END
                                FROM    breaks
                                JOIN    bn
                                ON      bn.place = breaks.place + size
                                LEFT JOIN
                                        top_rank
                                USING   (cluster)
                                )
                        SELECT  place, character, token IS NOT NULL
                        FROM    breaks
                        LEFT JOIN
                                top_rank
                        ON      1 = 1
                        CROSS JOIN LATERAL
                                (
                                SELECT  STRING_AGG(character, '' ORDER BY place) AS character
                                FROM    bn
                                WHERE   bn.place >= breaks.place
                                        AND bn.place < breaks.place + size
                                ) bn
                        WHERE   place > 0
                        )
                        )
                SELECT  place, character AS cluster
                FROM    bpe
                WHERE   NOT proceed
                ) bpe
        ),
        tokens AS
        (
        SELECT  ARRAY_AGG(token ORDER BY part_position, place) AS enter
        FROM    clusters
        JOIN    tokenizer
        USING   (cluster)
        ),
        gpt AS
        (
        SELECT  enter, ARRAY_LENGTH(enter, 1) AS original_length
        FROM    tokens
        UNION ALL
        SELECT  enter || next_token.token, original_length
        FROM    gpt
        CROSS JOIN
                enter
        CROSS JOIN LATERAL
                (
                WITH    RECURSIVE
                        hparams AS
                        (
                        SELECT  ARRAY_LENGTH(enter, 1) AS n_seq,
                                12 AS n_block
                        ),
                        embeddings AS
                        (
                        SELECT  place, values
                        FROM    hparams
                        CROSS JOIN LATERAL
                                UNNEST(enter) WITH ORDINALITY AS tokens (token, ordinality)
                        CROSS JOIN LATERAL
                                (
                                SELECT  ordinality - 1 AS place
                                ) o
                        CROSS JOIN LATERAL
                                (
                                SELECT  wte.values + wpe.values AS values
                                FROM    wte
                                CROSS JOIN
                                        wpe
                                WHERE   wte.token = tokens.token
                                        AND wpe.place = o.place
                                ) embedding
                        ),
                        remodel AS
                        (
                        SELECT  0 AS block, place, values
                        FROM    embeddings
                        UNION ALL
                        (
                        WITH    earlier AS
                                (
                                SELECT  *
                                FROM    remodel
                                )
                        SELECT  block + 1 AS block, transformed_layer.*
                        FROM    hparams
                        CROSS JOIN LATERAL
                                (
                                SELECT  block
                                FROM    earlier
                                WHERE   block < 12
                                LIMIT   1
                                ) q
                        CROSS JOIN LATERAL
                                (
                                WITH    ln_2_b AS
                                        (
                                        SELECT  *
                                        FROM    ln_2_b
                                        WHERE   block = q.block
                                        ),
                                        ln_2_g AS
                                        (
                                        SELECT  *
                                        FROM    ln_2_g
                                        WHERE   block = q.block
                                        ),
                                        c_proj_w AS
                                        (
                                        SELECT  *
                                        FROM    c_proj_w
                                        WHERE   block = q.block
                                        ),
                                        c_proj_b AS
                                        (
                                        SELECT  *
                                        FROM    c_proj_b
                                        WHERE   block = q.block
                                        ),
                                        mlp_c_fc_w AS
                                        (
                                        SELECT  *
                                        FROM    mlp_c_fc_w
                                        WHERE   block = q.block
                                        ),
                                        mlp_c_fc_b AS
                                        (
                                        SELECT  *
                                        FROM    mlp_c_fc_b
                                        WHERE   block = q.block
                                        ),
                                        mlp_c_proj_w AS
                                        (
                                        SELECT  *
                                        FROM    mlp_c_proj_w
                                        WHERE   block = q.block
                                        ),
                                        mlp_c_proj_b AS
                                        (
                                        SELECT  *
                                        FROM    mlp_c_proj_b
                                        WHERE   block = q.block
                                        ),
                                        c_attn_w AS
                                        (
                                        SELECT  *
                                        FROM    c_attn_w
                                        WHERE   block = q.block
                                        ),
                                        c_attn_b AS
                                        (
                                        SELECT  *
                                        FROM    c_attn_b
                                        WHERE   block = q.block
                                        ),
                                        ln_1_g AS
                                        (
                                        SELECT  *
                                        FROM    ln_1_g
                                        WHERE   block = q.block
                                        ),
                                        ln_1_b AS
                                        (
                                        SELECT  *
                                        FROM    ln_1_b
                                        WHERE   block = q.block
                                        ),
                                        mha_norm AS
                                        (
                                        SELECT  place, mm.values + c_attn_b.values AS values
                                        FROM    (
                                                SELECT  place, ARRAY_AGG(INNER_PRODUCT(c_attn_w.values, layer_norm.values) ORDER BY y)::VECTOR(2304) AS values
                                                FROM    (
                                                        SELECT  place, agg.values * ln_1_g.values + ln_1_b.values AS values
                                                        FROM    (
                                                                SELECT  place, norm.values
                                                                FROM    earlier
                                                                CROSS JOIN LATERAL
                                                                        (
                                                                        SELECT  AVG(worth) AS imply,
                                                                                VAR_POP(worth) AS variance
                                                                        FROM    UNNEST(values::REAL[]) worth
                                                                        ) agg
                                                                CROSS JOIN LATERAL
                                                                        (
                                                                        SELECT  ARRAY_AGG((worth - imply) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                                                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(worth, ordinality)
                                                                        ) norm
                                                                ) agg
                                                        CROSS JOIN
                                                                ln_1_b
                                                        CROSS JOIN
                                                                ln_1_g
                                                        ) layer_norm
                                                CROSS JOIN
                                                        c_attn_w
                                                GROUP BY
                                                        place
                                                ) mm
                                        CROSS JOIN
                                                c_attn_b
                                        ),
                                        heads AS
                                        (
                                        SELECT  place, head,
                                                (values::REAL[])[(head * 64 + 1):(head * 64 + 64)]::VECTOR(64) AS q,
                                                (values::REAL[])[(head * 64 + 1 + 768):(head * 64 + 64 + 768)]::VECTOR(64) AS ok,
                                                (values::REAL[])[(head * 64 + 1 + 1536):(head * 64 + 64 + 1536)]::VECTOR(64) AS v
                                        FROM    mha_norm
                                        CROSS JOIN
                                                GENERATE_SERIES(0, 11) head
                                        ),
                                        sm_input AS
                                        (
                                        SELECT  head, h1.place AS x, h2.place AS y, INNER_PRODUCT(h1.q, h2.ok) / 8 + CASE WHEN h2.place > h1.place THEN -1E10 ELSE 0 END AS worth
                                        FROM    heads h1
                                        JOIN    heads h2
                                        USING   (head)
                                        ),
                                        sm_diff AS
                                        (
                                        SELECT  head, x, y, worth - MAX(worth) OVER (PARTITION BY head, x) AS diff
                                        FROM    sm_input
                                        ),
                                        sm_exp AS
                                        (
                                        SELECT  head, x, y, CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
                                        FROM    sm_diff
                                        ),
                                        softmax AS
                                        (
                                        SELECT  head, x, y AS place, e / SUM(e) OVER (PARTITION BY head, x) AS worth
                                        FROM    sm_exp
                                        ),
                                        consideration AS
                                        (
                                        SELECT  place, ARRAY_AGG(worth ORDER BY head * 64 + ordinality)::VECTOR(768) AS values
                                        FROM    (
                                                SELECT  head, x AS place, SUM(ARRAY_FILL(softmax.worth, ARRAY[64])::VECTOR(64) * heads.v) AS values
                                                FROM    softmax
                                                JOIN    heads
                                                USING   (head, place)
                                                GROUP BY
                                                        head, x
                                                ) q
                                        CROSS JOIN LATERAL
                                                UNNEST(values::REAL[]) WITH ORDINALITY v (worth, ordinality)
                                        GROUP BY
                                                place
                                        ),
                                        mha AS
                                        (
                                        SELECT  place, w.values + c_proj_b.values + earlier.values AS values
                                        FROM    (
                                                SELECT  consideration.place, ARRAY_AGG(INNER_PRODUCT(consideration.values, c_proj_w.values) ORDER BY c_proj_w.place)::VECTOR(768) AS values
                                                FROM    consideration
                                                CROSS JOIN
                                                        c_proj_w
                                                GROUP BY
                                                        consideration.place
                                                ) w
                                        CROSS JOIN
                                                c_proj_b
                                        JOIN    earlier
                                        USING   (place)
                                        ),
                                        ffn_norm AS
                                        (
                                        SELECT  place, agg.values * ln_2_g.values + ln_2_b.values AS values
                                        FROM    (
                                                SELECT  place, norm.values
                                                FROM    mha
                                                CROSS JOIN LATERAL
                                                        (
                                                        SELECT  AVG(worth) AS imply,
                                                                VAR_POP(worth) AS variance
                                                        FROM    UNNEST(values::REAL[]) worth
                                                        ) agg
                                                CROSS JOIN LATERAL
                                                        (
                                                        SELECT  ARRAY_AGG((worth - imply) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(worth, ordinality)
                                                        ) norm
                                                ) agg
                                        CROSS JOIN
                                                ln_2_b
                                        CROSS JOIN
                                                ln_2_g
                                        ),
                                        ffn_a AS
                                        (
                                        SELECT  gelu.place, gelu.values
                                        FROM    (
                                                SELECT  place, w.values + mlp_c_fc_b.values AS values
                                                FROM    (
                                                        SELECT  ffn_norm.place, ARRAY_AGG(INNER_PRODUCT(ffn_norm.values, mlp_c_fc_w.values) ORDER BY mlp_c_fc_w.place)::VECTOR(3072) AS values
                                                        FROM    ffn_norm
                                                        CROSS JOIN
                                                                mlp_c_fc_w
                                                        GROUP BY
                                                                ffn_norm.place
                                                        ) w
                                                CROSS JOIN
                                                        mlp_c_fc_b
                                                ) v
                                        CROSS JOIN LATERAL
                                                (
                                                SELECT  place, ARRAY_AGG(0.5 * worth * (1 + TANH(0.797884560802 * (worth + 0.044715 * worth*worth*worth))) ORDER BY ordinality)::VECTOR(3072) AS values
                                                FROM    UNNEST(values::REAL[]) WITH ORDINALITY n (worth, ordinality)
                                                GROUP BY
                                                        place
                                                ) gelu
                                        ),
                                        ffn AS
                                        (
                                        SELECT  place, w.values + mlp_c_proj_b.values + mha.values AS values
                                        FROM    (
                                                SELECT  ffn_a.place, ARRAY_AGG(INNER_PRODUCT(ffn_a.values, mlp_c_proj_w.values) ORDER BY mlp_c_proj_w.place)::VECTOR(768) AS values
                                                FROM    ffn_a
                                                CROSS JOIN
                                                        mlp_c_proj_w
                                                GROUP BY
                                                        ffn_a.place
                                                ) w
                                        CROSS JOIN
                                                mlp_c_proj_b
                                        JOIN    mha
                                        USING   (place)
                                        )
                                SELECT  *
                                FROM    ffn
                                ) transformed_layer
                        )
                        ),
                        block_output AS
                        (
                        SELECT  *
                        FROM    hparams
                        JOIN    remodel
                        ON      remodel.block = n_block
                        ),
                        ln_f AS
                        (
                        SELECT  place, norm.values * ln_f_g.values + ln_f_b.values AS values
                        FROM    block_output
                        CROSS JOIN LATERAL
                                (
                                SELECT  AVG(worth) AS imply,
                                        VAR_POP(worth) AS variance
                                FROM    UNNEST(values::REAL[]) AS n(worth)
                                ) agg
                        CROSS JOIN LATERAL
                                (
                                SELECT  ARRAY_AGG((worth - imply) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n (worth, ordinality)
                                ) norm
                        CROSS JOIN
                                ln_f_b
                        CROSS JOIN
                                ln_f_g
                        ),
                        logits AS
                        (
                        SELECT  token, INNER_PRODUCT(ln_f.values, wte.values) AS worth
                        FROM    hparams
                        JOIN    ln_f
                        ON      ln_f.place = n_seq - 1
                        CROSS JOIN
                                wte
                        ORDER BY
                                worth DESC
                        LIMIT   (top_n)
                        ),
                        tokens AS
                        (
                        SELECT  token,
                                excessive - softmax AS low,
                                excessive
                        FROM    (
                                SELECT  *,
                                        SUM(softmax) OVER (ORDER BY softmax) AS excessive
                                FROM    (
                                        SELECT  *, (e / SUM(e) OVER ()) AS softmax
                                        FROM    (
                                                SELECT  *,
                                                        (worth - MAX(worth) OVER ()) / temperature AS diff
                                                FROM    logits
                                                ) exp_x
                                        CROSS JOIN LATERAL
                                                (
                                                SELECT  CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
                                                ) exp
                                        ) q
                                ) q
                        ),
                        next_token AS
                        (
                        SELECT  *
                        FROM    (
                                SELECT  RANDOM() AS rnd
                                ) r
                        CROSS JOIN LATERAL
                                (
                                SELECT  *
                                FROM    tokens
                                WHERE   rnd >= low
                                        AND rnd < excessive
                                ) nt
                        )
                SELECT  *
                FROM    next_token
                ) next_token
        WHERE   ARRAY_LENGTH(enter, 1) < original_length + threshold
                AND next_token.token <> 50256
        ),
        output AS
        (
        SELECT  CONVERT_FROM(STRING_AGG(SET_BYTE('x00', 0, byte), '' ORDER BY place), 'UTF8') AS response
        FROM    (
                SELECT  STRING_AGG(cluster, '' ORDER BY ordinality) AS response
                FROM    enter
                JOIN    gpt
                ON      ARRAY_LENGTH(enter, 1) = original_length + threshold
                CROSS JOIN LATERAL
                        UNNEST(enter) WITH ORDINALITY n (token, ordinality)
                JOIN    tokenizer
                USING   (token)
                ) q
        CROSS JOIN LATERAL
                STRING_TO_TABLE(response, NULL) WITH ORDINALITY n (character, place)
        JOIN    encoder
        USING   (character)
        )
SELECT  *
FROM    output