Langchain

intro

What are the uses of lang chain framework

• Build

• Run

• Deploy

• Maintain

• Monitour

What are some popular LLM architectures?

Examples include GPT (Generative Pre-trained Transformer), BERT (Bidirectional Encoder Representations from Transformers), and T5 (Text-to-Text Transfer Transformer).

LLM Task As per use case

Data Loader

This is done using the source File.

Main purpose is to load the file to the model

what is the purpose of env file

in that we have secreat keys like open api key & lot more.

for text file we use ______ loader

Document Loader

for Single pdf file we use ______ loader

pdf loaders or pypdf

for Multiple pdf file we use ______ loader

Directory Loader

for Reasearch Paper (arvix) file we use ______ loader

Document Loader

how do we verify the loader ?

we can verify the loader using refrence Object

Text Splitter

Spliting text as per the token limit.

Types of text splitters

character Splitter

Recursive Character Splitter

Recursive Json Splitter

Character Splitter

/n ← split by next line

/n /n ← Split by paragraph

Chunks

Smallest part of main data for token limit

Chunk Overlap

This will help in maintaining the context of the Info.

Cosine similarity used to determine the result by

Cosine Similarity=A⋅B/∥A∥∥B∥

The result ranges between −1-1 and 11, where:

• 11 means the vectors are identical.

• 00 means they are orthogonal (completely different).

• −1-1 means they are opposite

Indexing

Once Chunk is Created that is converted to vector with meaning.

if Chunk Capacity is 4k Tokens can we send 2k tokens

Yes

Recursive Character Splitter

This will split as per the sentance. This has the context of the words

Recursive Json Splitter

we will load the data from json link and we will use context manager to store File here.

do we send full Token

we only send 75 % of the token info

if your model token limit is 4k and you send documnet with 5k what happens to last1k data ?

That wont even read.

What is the challenge of handling token limits in LLMs?

LLMs have maximum token limits for input and output, which can make it challenging to process or generate long-form text efficiently.

How do LLMs handle context in a sentence or passage?

LLMs use attention mechanisms, such as in the Transformer architecture, to focus on relevant parts of the input sequence and understand context.

How would you choose between PyTorch and TensorFlow for training LLMs?

PyTorch is more flexible and widely used in research, while TensorFlow is production-friendly and integrates well with deployment tools.

what is a prompt types

system and custom prompt

how do you code Prmpt ?

prompt = PromptTemplate(
    input_variables=["product"],
    template="What is a good name for a company that makes {product}"
)

what is template and place hodler in prompt

parameters that control the performance and behavior of models

Learning Rate (for training or fine-tuning)

Context Window (Token Limit)

Batch Size

Stop Sequences

Frequency Penalty

Presence Penalty

Max Tokens

Top-k

Top-p (Nucleus Sampling)

Temperature

Explain the parameters :

• Temperature:

  • This parameter controls the randomness of the model's responses.

  • Low values (e.g., 0.2) make the output more focused and deterministic. The model tends to pick the highest probability answer every time.

  • High values (e.g., 0.8) make the output more diverse and creative, as it allows the model to explore less probable options.

• Top-p (Nucleus Sampling):

  • It controls how many probabilities the model considers when generating each word or token.

  • For example:

    • Top-p = 1.0: The model considers all probabilities (like traditional sampling).

    • Top-p = 0.9: The model only considers the top 90% of probable choices and ignores the rest, leading to more focused yet flexible output.

• Top-k:

  • This limits the number of top choices for each word the model considers.

  • Low values (e.g., k = 10) constrain the model to select from only the top 10 options, making the responses more predictable.

  • High values allow greater exploration but can lead to randomness.

• Max Tokens:

  • This parameter sets the maximum length of the output generated.

  • Increasing this parameter allows the model to generate longer responses.

  • Reducing it limits the output length for shorter answers.

• Presence Penalty:

  • Controls how much the model discourages repeating words or ideas it has already generated.

  • High penalty results in more diverse responses by avoiding repetitions.

  • Low penalty allows frequent repetition if required.

• Frequency Penalty:

  • Similar to presence penalty but specifically discourages generating frequent words (e.g., common verbs).

  • Encourages more creative or novel responses.

• Stop Sequences:

  • Specific sequences of words can be defined to stop the model's output generation.

  • For example, you might set a stop sequence like "End of conversation," and the model will cease generating text after encountering it.

• Batch Size:

  • Controls how many predictions the model processes at once during training or inference. Larger batch sizes can speed up processing but require more memory.

• Learning Rate (for training or fine-tuning):

  • Determines how much the model adjusts its weights during each iteration of training. A higher learning rate speeds up training but risks overshooting optimal values, while a lower rate provides stability but slows down the process.

• Context Window (Token Limit):

  • Defines the maximum number of tokens the model can process in a single prompt. Larger context windows allow the model to understand longer inputs or conversations.

Chain

This will tell you order of Execution

if we have Multiple chains we use

Chain Channel

Chain Types

Simple Sequential Chain
Sequential chain

Simple Sequential chain

we want output of last chain

Sequential chain

This will give Output of all chain.

how do we see how chain is performing

Verbose is True

Agents in LLM

This will intract with LLM using

Prompt, task automation & Context handling

Tools in LLM

Run the query ex : external apis are used to perform external tasks.

memory in LLM

Help model to remember the previous chat (Conversation)

Conversation Memory in memory

This remember all the conversation and print only place holders values.

Conversation chain in memory

You get both Q& A

Conversation Buffer window Memory in memory

This will remember to certain number of conversation as per specified k (Context Window).

Embedding

This help converting text to number as model cannot understand other than number

Types of Embedding

Open Ai , Hugging Face , Ollama , BERT, TF IDF

Vector Index

This will store the embedding in your own system.

only for simple task not Company Related, only used for testing

ex FAISS

Vector DB

The embedding are stored in cloud

We mostly use them for Company

Ex Chroma ,Weaviate & pinecone

Meta data in Vector Data

This is Key Value Pair of vector.

we use this for storage limit, Character limitation & set max 50 per index.

How do we know a particular chunk has our related info

similarity Search (Cosine similarity , Man hat , Euclidean distance & Approximate nearest nebhour)

why we don’t use the SQL as vector db. ?

A custom text file has millions of texts; when we convert it, the latency will be low.

Retrieving using sql will have very bad latency as it’s not a vector db

Faiss full form

Facebook AI Similarity Search

FAISS

FAISS is widely used in applications like recommendation systems, search engines, and Large Language Model (LLM) indexing due to its ability to handle large-scale datasets with high performance.

Vector Index

No scalability

We need backup

what is dangerous serialization = true

you trust the dock you send to model.

earlier we used to not use this. but now this has become manditory.

Vector Db Types

Chroma , Weaviate & pine cone

Chroma

This is self hosted db. ITs from Apache.

Used for long term Retention.

Weaviate

Open source with Plugin and extensibility

self Hosted or Cloud

PineCone

This is used for cloud only.

Used mostly for scaling data sets.

Rag Full form

Retrieval augmented Generation

RAG work

retriever : this will find relevant info from query (input)

Augmentation : This takes relevant info from input to llm

Generate : Takes input of retriver & augmented to generate the output

Why we use a rag

As LLM cannot correctly predict from external data, we use rag.

This will remove Hallucination as this provides info that is not available to the model.

Finetuning Vs rag

Aspect	Fine-tuning	RAG (Retrieval-Augmented Generation)
Definition	Updates model with domain-specific data.	Combines model with external data retrieval.
Training Required	Requires retraining for new tasks.	No retraining needed; uses external sources.
Real-time Updates	Difficult to incorporate.	Easily integrates updated or live information.
Inference Speed	Faster, no external dependencies.	Slower, involves retrieving external data.

FineTunning

We can train a pre existing model and make that as per our requirements.

You should have Good GPU to use Fine Tuning.

FineTunning

LORA ,Qlora , Quantization

Model Compression

This will reduce the size of the model & technically complexity of model

drawback in model compression

Loss of accuracy

Types of model compression

quantization

pruning

weight sharing

knowledge distillation

Knowledge distillation

Train small model to mimic the behaviour of big model

Pruning

removing unnesary model parameters like weights so size of model is reduced

weight sharing

Layer of neuro share the weight thus reducing the size of model

Quantization

Quantization in LLMs (Large Language Models) is a technique used to make models smaller and faster by reducing the precision of the numbers they use to perform calculations.

Example of quantization

32 Bit FP → 8 bit FP

FP : Full Precision

Ex 1.1311341 → 1.1 FP

LORA vs QLORA

Aspect	LoRA (Low-Rank Adaptation)	QLoRA (Quantized LoRA)
Purpose	Efficient fine-tuning of pre-trained models.	Efficient fine-tuning combined with quantization to further reduce memory usage.
Technique	Injects low-rank matrices into model weights during training without updating the full model.	Applies LoRA fine-tuning on models quantized to 4-bit precision for extreme resource efficiency.
Memory Usage	Low memory requirements compared to full fine-tuning.	Significantly lower memory usage due to quantization (ideal for large-scale models).
Precision	Retains floating-point precision (e.g., FP32/FP16) during fine-tuning.	Reduces precision to 4-bit quantization while maintaining performance with minimal accuracy loss.
Performance	High fine-tuning efficiency, with minimal accuracy trade-off.	Comparable performance to full-precision models, despite quantization.
Computational Cost	Reduces cost by avoiding the need to train full model weights.	Further reduces computational cost by leveraging quantized models

Rag Types

Dense retriver : used for vector retriver Ex BERT

Sparse Retriver : Utilization like TFIDF

Hybrid : dense + Sparse

Multihop : Retriver From Multiple Steps

Personalized Retriver : Retries Based on History

How do we version Controll

We will use GIT

RAGAS

This is used to evaluate the accuracy of RAG

• faithfulness

• Precision

• Recall

• Correctness
  

Model scaling law

Model perfomance will depends on GPU

Model perfomance will decrease when the size of model increase

Halucination

Incorrect answer is given by model that is grammatically correct but not actually correct.

how do we reduce the halucination

Rag , reinforment learning , prompting , model calibraion ,Fact Verification and ensambling techniques

RAG in halusination

this depends on external data apart from internal data

Prompting in hallucination

Good prompts also reduce the hallucination

model calibraion

Fact verification

Help model to verify the facts in online

ensambling techniques

Combine output of multiple models to reduce hallucination

LLm evaluation

To Check the perfomance of LLM

Perplexity , ragas , BELU , Human , meteor

Perplexity:

• Think of it as how "confused" a language model is when predicting the next word in a sentence.

• Lower perplexity means the model is better at guessing correctly and is more confident.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation):

• It's a way of comparing text generated by the model with reference text.

• It checks for overlapping words and phrases between them to measure how close they are.

• Widely used in summarization tasks.

BLEU (Bilingual Evaluation Understudy):

• A metric to evaluate how well machine-generated text matches human-written text.

• It looks at word sequences (like pairs or triples) and rewards similarity to the reference text.

• Commonly used in translation tasks.

Human Evaluation:

• Sometimes, machines just can't do justice! Humans read the model's outputs and judge them based on quality, fluency, and relevance.

• It's subjective but gives insight into how "natural" the model's responses are.

METEOR (Metric for Evaluation of Translation with Explicit Ordering):

• Similar to BLEU, but smarter about matching synonyms and different word orders.

• It's designed to capture the meaning better than just exact word matches.

How do you choose The LLm Model

its said by client as per budget or some time we suggest him better option if available

using llm evalutaion we check which model is performing good and later decide to select that model

for open source GPU and storage must be good

which is costly open source or non open source

non open source (GPT) but open source (LLama)has storage cost

which model have privacy concerns

Open source

Easy to change Things is model

non open source is easy to change the model as we only change the model name buy in open source we need to make a lot of changes

Multi stage environment

Data ingestion , Chunking , embedding , Retrieval , Generation of summary

Context Learning

The model is learned by based on the context provided by the model.

we provide example for what type of answer we want.

security Features in Azure Open Ai

Content filter (remove Hateful media), data encryption & windows defender for cloud.

PII management without exposing to end user :

Encryption data is stored

Role based access

Strong Authentication

monitor unauthorised acess