BDS 3 LLM

What are the 8 Arguments by Häggström why LLMs are intelligent.

1. Lack of common sense

Anyone who has said something dumb would be "unintelligent" by this logic — including every human.

2. Only trained facts

Humans only know facts from experience or evolution. Both humans and LLMs can derive new facts via inference.

3. Just matrix multiplication

Human brains are "just atoms" — intelligent systems can be built from unintelligent components.

4. Only predicts next word

Confuses training objective with emergent behavior. Evolution "trained" humans to reproduce — yet humans do far more.

5. No world model

LLMs exhibit world-model-like behavior. Humans rely on introspection — but introspection can only reveal one's own mind, not serve as a universal test.

6. No symbol grounding

Kant: humans also lack direct access to things-in-themselves. All experience — human or LLM — is mediated by signals (neurons vs. text).

7. No creativity

Turing's rebuttal of Lovelace (1950): if we credit only external causes, human creativity is also impossible. Creativity = producing something unanticipated.

8. No consciousness

Intelligence (what you can do) ≠ consciousness (what it feels like). We can't prove other humans are conscious either.

"None of the above rules out that LLMs lack something crucial to high-level intelligence… but the most widely circulated arguments for that being the case all fail to convince." — Häggström

Neural network mechanics (Basic Neural Network) (Universal approximation theorem)

Basic neural network

Layers of nodes with weighted connections. Each node applies a nonlinear activation function. Weights are learned from data.

Universal approximation theorem

A neural network with sufficient hidden units can approximate any continuous function — the theoretical basis for why NNs are so powerful.

4 Steps to build a neural network

1. Counting

2. Adding

3. Multiplying

4. Predicting

These primitives compose into neural networks. LLMs predict the most probable next token given context — but emergent behavior goes far beyond simple autocomplete.

Difference between How AI works and What AI can do

How AI works

The underlying mechanics — matrix math, weighted sums, activation functions. Like a pile of Lego bricks

What AI can do

The emergent capabilities — reasoning, translation, prediction. Like an assembled Lego figure.

What are the three types of word embeddings?

1. Basic embedding

Each word gets a fixed vector from a lookup table (e.g., "Meryl" → [3, -1.8]). No context-sensitivity.

2. Skip-connection embedding

Numbers are passed forward across layers, allowing deeper representations to retain earlier information.

3. Attention-based embedding

Each token's representation is updated based on other tokens — "price" in "won the price" vs. "price of milk" gets different vectors.

How does the Attention mechanism (the “long answer”) work? Explain query, key and value

Each token has a query, key, and value vector. The query of one word is compared against keys of all other words (via dot product) to produce attention weights. The output is a weighted sum of value vectors. This lets the model relate distant tokens — e.g., resolving that "price" in "Meryl won the price" relates most strongly to "won".

Example from slides: price = 0.6 × "won" + 0.3 × "price" + 0.02 × "home" …

For the phrase "Meryl won the price and went home", when the model processes the word "price," it needs to figure out which surrounding words carry relevant context. Its query vector gets compared against the keys of every other word. "Won" gets a high match (key ≈ query), so its value gets a weight of 0.6 in the output. "Price" itself gets 0.3, "home" gets 0.02, and so on:

The resulting vector for "price" now carries contextual information — it "knows" it's in the context of winning something, not a price tag in a store. That's the magic: the same word gets a different representation depending on its context.

Query: Topic or Question I’m interested in. (Movies about war)

Key: Short description like (war drama)

Value: Content of the book

Important key (what the book advertieses) and value (what it contains) can be different things.

Multi-head attention simply runs this process several times in parallel with different learned Q/K/V matrices, letting the model attend to different types of relationships simultaneously — one head might track syntactic structure, another semantic meaning, another coreference, and so on.

How does Basic Embedding work? Why do we need it?

Otherwise, an insane amount of computation?

First of all we think of a neural network. There is a difference between a multi-word context and a single-word context. The neural network changes the prediction based on the context of the words it has. But certain similar words (Meryl and Clooney) lead to similar predictions.

The two hiddin neurons hold the numerical values 3 and -1,8. These are mathematical representations of the word Meryl.

The embedding process can be thought of as a simple dictionary look-up book when the model encounters the word "Meryl", it simply flips open the book and retrieves its multi-dimensional coordinate vector.

How does Positional encoding work? And why do we need it?

Because a transformer looks at all words in a sentence simultaneously. There is no concept of word order.
To a Transformer, the sequence "Meryl won the award" looks exactly the same as "The award won Meryl".

How it works:

1. Each position gets a unique vector.
- Every slot in a sequence gets its own vector (1st word, 2nd word, 3rd word ….)

In [0.8, -0,5] just represents the 4. Position in a sentence

2. Fixed Wave Patterns

Why use waves instead of numbers? → Bc if sentences get incredibly long, the model can get confused.
So instead the transformer uses sine and cosine waves to generate these vaules

Look at Token Position 4 on the x-axis, you can read the corresponding y-axis values on the curves. Those exact coordinates give you the unique identifier

So now we have the meaning of a word (Embedding) + Position of the word (Positional encoding)

These two just get added together.

How does a Transformer work?

The full transformer (as shown in the slides) has an encoder (reads input, builds contextual representations using multi-head attention + positional encoding) and a decoder (generates output token-by-token using masked attention + cross-attention to encoder). Each block contains: multi-head attention → add & norm → positionwise FFN → add & norm. The "basic NN" (FFN) appears inside every transformer block.

What is a Encoder useful for what is a Decoder useful for?

Encoder: Good for Embeddings, Classification (e.g., BERT)

Decoder: Good for Text generation (e.g., GPT)

What is meant by Sources and what comes out at the FC (Decoder)

Sources: Meryl Streep won another

FC(Decoder): Oscar

What are skip-connections?

Numbers are passed forward across layers, allowing deeper representations to retain earlier information.

How does a basic Neural Network work? (1. structure)

1. Structure

• Input Layer: This is where the data enters the network. Each neuron here represents a feature of your data (for example, in the embedding slides you shared earlier, each input neuron represented a specific word like "Meryl", "the", or "won").

• Hidden Layer(s): These intermediate layers do the heavy lifting. They extract abstract patterns from the input data.

• Output Layer: This produces the final prediction (e.g., a probability of 75% that the next word is "Oscar").