LLM Knowledge

Topics Covered

  • Ways to improve latent models without updating weights.

  • Large language models and knowledge bases.

  • Evaluating large language model knowledge.

  • Adding language model knowledge.

Self-Evolving AI Development (May 2025)

  • Rapid development with multiple papers discussing iterative optimization of code or training.

  • Potential concerns regarding AI improving itself.

"Darwin Godel Machine"

  • Agent (Ada) can read and modify its own codebase.

  • Explores new design spaces and evaluates itself on benchmarks.

  • Modifies its own code based on feedback.

"Self-training Large Language Model (LLM) frameworks like Ada enable continuous learning, allowing the model to refine its algorithms through iterative processes, ultimately improving performance over time. "

  • Unsupervised self-rewarding training.

  • Rewards alignment across generations (similar to self-consistency prompt engineering).

  • Comparable performance to standard reinforcement learning in early stages.

  • Training collapse and nonsense answers in difficult datasets, especially in larger models.

Large Language Models as Knowledge Bases

  • Potential for using large language models as knowledge bases.

  • Evaluating, adding, and updating knowledge within language models.

Prompt Engineering Review

  • Natural language descriptions are sufficient for larger language models.

  • Improving language models through prompts without updating model parameters.

Magic Word

  • Zero-shot prompting.

  • Importance of step-by-step immersion and emotional stimuli.

  • Using direct linear language, adding tips, and penalties.

  • Automatic prompt engineering to search for effective phrases.

  • Example: "Let's work this out in a step-by-step way to be sure we have the right answer" is better than just saying, "Step by step."

  • Magic word effectiveness may decrease over time as models improve.

Feeding More Context

  • Specifying personal context makes a difference.

  • In-context learning: larger models can learn from examples in the context.

  • Few-shot prompting: providing generative models with question-and-answer examples.

  • Adding reasoning to examples to prompt models to provide reasoning.

Monitor Prompt Engineering

  • Task decomposition: breaking down tasks into multiple steps and sub-questions.

  • Knowledge generation first.

  • Self-reflection: models improving and checking their own answers.

Tool Usage

  • Improving larger language models without weight updates.

  • ChatGPT as an agent-like interface can search online, code, and draw figures.

  • Using DALL-E to output pictures for creative painting or diagrams.

System Integration

  • Larger models can be linked within systems.

  • Using a database to retrieve information based on user prompts.

  • Combining retrieved data with prompts for the larger model.

Retrieval-Augmented Generation (RAG)

  • Generation augmented with information retrieval, proposed by Google.

  • Supplementary database for larger models.

Dense Passage Retrieval (DPR)

  • Using two encoders: one for user query and one for document.

  • Finding relevant documents by comparing semantic similarity rather than keywords.

Fusion in Decoder (FID)

  • Encoder deals with each query and document separately.

  • Encodes query and each document into embeddings.

  • Combines embeddings using an attention mechanism to assign importance scores to each document.

  • Sends combined information to larger models, allowing them to reason across documents.

Tool Users

  • Systems, not larger models, control the process by using tools to supplement generation.

Multi-Agent Systems

  • Multiple large language models collaborating on a single problem like a team.

Debate Example

  • Counterintuitive question: circle A revolves around circle B.

  • Models debate to find mistakes and reason.

  • A acts like it circles around B plus it circles around its own perimeter. Therefore, it circles a total of 4 times around B.

  • Final judge determines the answer.

  • More models lead to better debates, reducing disagreement and increasing accuracy.

Multi-Agent Collaboration Paradigm

  • Dynamic large language model agent network for tasks like software development.

  • Roles: CEO (human), program managers, testers, programmers (all direct models).

  • Software development cycle: planning, coding, feedback, iterative improvement.

Hyper-fixation Prevention

  • Early stopping to prevent collapse.

  • Using pseudo labels in self-supervised learning to constrain model development.

Large Language Model Knowledge

  • Guiding incremental behaviors with prompts.

Internal Knowledge

  • Encoding factual, linguistic, and relational knowledge across billions of parameters.

  • Using models as a knowledge base.

Knowledge Base Definition

  • Repository of information for retrieval or decision-making.

  • Offers ground truth data represented by structured knowledge graphs (nodes and edges).

  • Nodes: Entities

  • Edges: Relationships between entities

Wikidata Example

  • Knowledge graph extraction with person's name, date of birth, and relationships.

Knowledge Base Query

  • SQL-like queries to select information.

  • Example: Select date of birth from person named Franklin.

Building Knowledge Base

  • Raw data from unstructured text (internet, newspapers, books).

  • Knowledge extraction and formatting to structure the data.

Shortcomings

  • Knowledge extraction complexity involving multi-step natural language processing pipelines.

  • Human labeling and annotations are needed.

  • Mistakes propagate to later stages.

  • Information loss during the extraction process.

Language Models as Knowledge Bases

  • Prompts are the primary risk.

  • Asking questions to models to retrieve their information.

Advantages

  • Pre-trained on massive general-purpose text corpora, scalable.

  • Almost no human annotations are needed.

  • Flexible and easily queried with natural language prompts.

  • Accessible, easy to copy, and easy to deploy locally.

Verifying Knowledge

  • Asking factual and common sense questions.

  • Comparing results with the ground truth to assess accuracy.

LAMA (Language Model Analysis)

  • Benchmark to evaluate factual knowledge encoded by language models, introduced by Amana.

  • Fill-in-the-blank questions about factual information and knowledge bases like Wikidata.

  • Format for BERT using a mask special token.

  • Language model predicts the next token.

Example

  • "The color of the sky is ___," where "blue" is masked.

  • Model assigns probabilities to all vocabulary, with the highest probability expected for "blue."

Evaluation

  • Precision at K: whether the correct token appears in the top K of the ranked results.

Pipeline

  • Gather factual knowledge from knowledge bases and compile them into question-answer pairs.

  • Create close statements with masked tokens using templates.

  • Use the LAMA benchmark to test the knowledge of a language model.

Discussions about Language Model Knowledge

Knowledge Parroting

  • Distinguishing true knowledge from memorized patterns.

ECL 21 Paper

  • Looked at the overlap of questions and answers between training and testing data.

  • 60-70% overlap of answers and 80% overlap of questions in most free datasets.

  • Knowledge retrieval is highly boosted after seeing similar questions or answers.

Are They Knowledgeable or Simply Guessing?

ECL 21 Paper Revisiting LAMA Benchmarking
Limitations

  • Prompt bias: Rewriting or paraphrasing prompts changes retrieval performance.

  • Type Guessing: Models can guess the type of answer (noun, location, name) from context.

  • Context Influence: Helpful, neutral, or misleading context affects performance.

Real World Differences

  • Language model knowledge is not true life.

Reporting Bias

  • Text data is from the internet, which are noteworthy events.

  • Overestimates significant events and underestimates common everyday occurrences.

Callington T Paper

  • Tracked reporting bias in language models.

  • Early language models preferred words associated with significant events like "killed" or "married."

Knowledge Editing for Language Models

  • Focus on outdated text corpora.

  • Updating specific factual knowledge without retraining models from scratch.

Challenges

  • Knowledge is hard to define.

  • Multiple questions can be centered around one piece of knowledge.

Knowledge Neurons

  • Related to explainable artificial intelligence.

ACL 2022 Paper

  • Certain neuron activations are related to factual knowledge.

  • Adding knowledge by identifying and changing specific neurons.

Process

  • Human efforts:

    • Designing different prompts to query target fact.

    • Identify overlapping neurons between prompts (neurons responsible for that knowledge)

    • Suppress the activation corresponding to outdated facts such as old Prime Minister's name.

    • Amplify the activations of relevant neurons corresponding to the new knowledge.

Next Lectures

  • Training and fine-tuning of language models.

  • Topics: pre-training, instruction fine-tuning, reinforcement learning from human feedback, PPO, DPO, and more.