CPMAI

Artificial Intelligence is defined as

Machine behavior and function that exhibit the intelligence and behavior of humans.

CPMAI Six Phase Methodology

1. Business Understanding — Define the problem, evaluate AI fit, set KPIs and Go/No-Go criteria

2. Data Understanding — Identify data needs, assess quality, availability, and data roles

3. Data Preparation — Clean, label, transform, and engineer data for model training

4. Model Development — Select algorithms, train, tune, and validate the model

5. Model Evaluation — Test against business and technical KPIs; determine readiness for deployment

6. Model Operationalization — Deploy, monitor, manage, and iterate the live AI system

Seven Patterns of AI?

1. Conversational & Human Interaction: Machines interact with humans using natural language (voice, text, images). Examples: chatbots, voice assistants, content generation, sentiment analysis, machine translation.

2. Recognition: Machines identify/understand real-world unstructured data. Examples: facial recognition, image classification, handwriting ID, object detection, gesture recognition.

3. Predictive Analytics & Decision Support: Uses ML to predict future outcomes from past/existing data. Supports human decisions — does NOT remove the human from the loop. Examples: dynamic pricing, equipment failure prediction, fraud detection, demand forecasting.

4. Goal-Driven Systems: Uses reinforcement learning to find optimal solutions through trial and error. Examples: game playing, resource optimization, robo-advising, bidding/auctions, simulations, iterative problem-solving.

5. Hyper-Personalization: Develops a unique, evolving profile of each individual using ML. Examples: personalized healthcare treatments, personalized finance plans, personalized education/training, behavior profiling.

6. Autonomous Systems: Physical and virtual systems that accomplish goals with minimal/no human involvement. NOT the same as automation. Examples: autonomous vehicles, autonomous robots, autonomous software systems.

7. Patterns & Anomalies: Identifies patterns in data and anomalies that deviate from those patterns. Examples: fraud detection, outlier detection, cybersecurity, predictive time-series analysis, content summarization.

What are the Three Ps of intelligent systems?

- Perception (sensing and processing the environment)

- Prediction (foreseeing what might happen next)

- Planning (acting based on perception and prediction).

Together they form an intelligent feedback loop

What is the difference between Automation and Autonomous?

• Automation handles repetitive, programmed, predictable tasks (it follows defined rules).

• Autonomous means a system can perform dynamic, complex tasks with minimal or no human involvement, using intelligence to handle variability (think of a Waymo vehicle).

What is Narrow AI?

Narrow AI handles only a single or small set of related tasks for a specific function. All current AI is Narrow AI (we are not yet at Artificial General Intelligence (AGI).

What is Algorithmic Explainability (XAI)?

The ability to understand and explain WHY a model arrived at a specific prediction or decision. Required by law in many regulated industries. Deep learning models are often NOT explainable (black box), making XAI compliance a Phase I constraint to identify early.

Trustworthy AI: 5 Layers

1. Ethical AI: Societal values — dignity, fairness, human agency

2. Responsible AI: Laws and regulations — legal compliance, prevent misuse

3. Transparent AI: Systemic visibility — how built, bias measurement, disclosure

4. Governed AI: Practices and processes — audits, controls, contesting mechanisms

5. Interpretable/Explainable AI: Technical — explains individual predictions

The Four V’s of Big Data

• Volume: The challenge of dealing with very large amounts of data, often spread across different locations.

• Velocity: The challenge of data that is rapidly changing or moving, requiring processing at high speed with necessary accuracy. Streaming data, stock prices, sensor data from aircraft.

• Variety: The challenge of data in different formats, from different locations, and with varying levels of structure: structured, unstructured, and semistructured.

• Veracity: The challenge of data with varying levels of quality, accuracy, trustworthiness, and consistency. Problematic especially at scale.

AI Data Types: The 10/80/10 Rule

• Structured (~10%): Data with a defined format and rigid schema (rows and columns). Examples: relational databases, spreadsheets (CSV, Excel), SQL, Parquet, ERP/CRM systems. Key Characteristic: Easiest to work with. Standard BI tools and SQL queries apply. Easy to extract specific fields.

• Unstructured (~80%): Data without a predefined schema — highly variable even within a single domain. Examples: images, videos, audio, emails, PDFs, social media posts, text documents. Key Characteristic: Requires specialized tools (ML, text/vision analytics). Cannot be queried with SQL. Holds the most untapped organizational value.

• Semistructured (~10%): Data with partial structure via tags or metadata, but with variable content. Examples: JSON, XML, HTML, NoSQL databases, invoices, system log files. Key Characteristic: Requires parsers or NoSQL queries. Has elements of both structured and unstructured data.

Data Steward / Data Engineer / Data Scientist / Data Custodian

• Data Steward: Primary Responsibilities: Ensures data is accessible, trustworthy, usable, and secure. Enforces data governance policies. Manages data quality, lineage, cataloging, and monitoring. Serves as data advocate. Key Distinguisher: Governance/quality/access role. Has both technical AND soft skills. Works across IT and business units. Owns data lineage documentation.

• Data Engineer: Primary Responsibility: Builds and maintains data pipelines, data ingestion systems, ETL processes, and data infrastructure. Focuses on technical movement, transformation, and storage of data. Key Distinguisher: Technical pipeline role. Moves data from source to destination. Builds the infrastructure the data scientist and AI model depend on. Most of the 80% data engineering effort lives here.

• Data Scientist: Primary Responsibility: Extracts analytical insights from data. Builds and applies ML models. Translates business needs into mathematical/statistical approaches. Tests hypotheses using algorithms. Key Distinguisher: Analytical modeling role. Works on classification, regression, clustering, and prediction. Focuses on insight and model quality, not pipeline infrastructure.

• Data Custodian: safe storage, transfer, and administrative management

Data Stewardship Data Governance, Data Quality Management, Data Custodianship

• Data Stewardship: The PRACTICE of ensuring the organization's data is accessible, trustworthy, usable, and secure. Focuses on the HOW — ensuring governance policies are actually enforced and followed. Encompasses the full data life cycle: collecting, transforming, using, storing, archiving, deleting.

• Data Governance: The set of processes, procedures, standards, roles, and tools an organization implements to ensure data is properly stored, managed, accurate, available, secured, and controlled over its life cycle. Deals with the WHAT and WHY — defining the policies, rules, and accountability structures. Addresses: security risks, privacy risks, data ownership, auditing, access control, data sharing, compliance.

• Data Quality Management: the process of improving and maintaining information used for analysis and decision-making. It is one of the types of data management — focused specifically on accuracy, completeness, and reliability of data.

• Data Custodianship: safe storage, transfer, and administrative management

3 Meanings of Bias in AI

1. Neural network bias — a mathematical adjustment factor in model weights.

2. Bias-variance trade-off — model tendency to underfit or overfit.

3. Informational bias — overrepresentation/underrepresentation in data.

Types of Analytics - 4 Categories

• Descriptive Analytics: 'What happened?' Focuses on understanding historical data, relationships, and comparisons. NOT forward-looking. Uses reporting tools, charts, summaries. Best for deterministic, historical reporting.

• Diagnostic Analytics: 'Why did that happen?' Focuses on identifying cause-and-effect relationships. Explores root causes of historical events. More sophisticated than descriptive.

• Predictive Analytics: 'What could happen?' Uses past and current data to make predictions about future or unknown events. Requires ML models and data science techniques.

• Prescriptive/Projective Analytics: 'What if?' Focuses on identifying the potential impact of decisions based on current data. Simulates scenarios and recommends optimal actions. Most sophisticated analytics type.

What is Training Data?

A data set of prepared, cleaned, and appropriately labeled data (if used for supervised learning) that is used to incrementally train an ML model to perform a particular task. It must be representative of real-world conditions to produce a reliable model.

What is Ground Truth Data?

Training data gathered from real-world sources, reflecting actual users, transactions, and events. Key challenges: incompleteness, need for labels/annotation (especially for supervised learning), format and cleanliness issues, accuracy problems, labeling effort, and privacy/confidentiality concerns.

What 3 core aspects of data must be addressed in Phase 2 (Data Understanding)?

1. Data Sources: where does the data come from, who owns or stewards it, and how is it accessed?

2. Data Description: what is its structure, type, and format — structured, unstructured, or semistructured?

3. Data Quality: how clean, complete, and representative is it?

Phase 2 (Data Understanding) Go/No-Go Criteria?

You can advance to Phase III when:

1. Data requirements have been adequately determined

2. Adequate workbook responses exist for Phase II items

3. No critical roadblocks prevent project success.

You do NOT need final answers — but you must know what data you have, where it is, and how to access it.

80/20 Rule

80% of an AI project's total effort is data engineering (Phases 1-3), and only approximately 20% is model development (Phases 4-6). This has direct consequences for team composition and resource planning.

• Implication for Team Composition: You need significantly MORE data engineers than data scientists on an AI project team. Overstaffing data scientists and understaffing data engineers is one of the most common misallocation mistakes.

• Implication for Project Planning: Most of the time, budget, and effort is spent BEFORE the model is ever trained. Phase III alone (data preparation) is where the majority of hands-on work occurs.

• Implication for Sponsors: Sponsors who expect quick model results underestimate the data engineering investment. CPMAI requires communicating the 80/20 reality to stakeholders at Phase I.

• The Exam Trap: GenAI projects do NOT reduce the 80/20 rule. CPMAI states that GenAI projects often require MORE data management intensity, not less — foundation models still need domain-specific data preparation, prompt engineering data curation, and RAG infrastructure.

Data: Splitting / Augmentation / Multiplication / Transformation / Labeling

• Data Splitting: Separating data into training, validation, and test sets. Includes data sampling for very large data sets and data attribute pruning to reduce size and complexity.

• Data Augmentation: Enhancing existing data QUALITY with additional manipulations or combinations to increase effective quantity. Adds necessary information to the original data. Especially important for unstructured data (text, images, video, audio).

• Data Multiplication: Increasing the QUANTITY of prepared data by transforming and manipulating existing data. Example: starting with 1,000 images and generating 100,000–200,000 through transformations. Distinct from augmentation (which adds quality/information); multiplication adds volume.

• Data Transformation: Changing data from one state to another — converting format, altering metadata. Part of the ETL/ELT process to change data into the right format for storage or model training.

• Data Labeling: Adding descriptive tags or metadata to training data — especially for supervised learning. Provides meaning so models can learn by example. Without labels, supervised learning cannot proceed.

ETL vs. ELT - Data Pipeline Approaches

• ETL (Extract, Transform, Load): Traditional method for loading data into DATA WAREHOUSES. Transform step occurs BEFORE loading because data warehouses require data in a specific format. Steps: Extract from source → Transform (merge, combine, convert) → Load into warehouse. Best for: structured data, relational systems, when data must conform to a fixed schema.

• ELT (Extract, Load, Transform): Modern method introduced with DATA LAKES. Data is loaded as-is first, then transformed when needed for specific tasks. Steps: Extract from sources → Load into data lake → Transform on demand. Best for: big data environments, diverse formats, when flexibility is needed and transformation can be deferred.

Move Data to Processing vs. Move Processing to Data?

• Move Data to Processing: Used when data is NOT too large AND does not change frequently. Export the data from its source to the processing location. Methods: manual export, automated export, replication.

• Move Processing to Data: Used when data IS too large to move OR changes too quickly to keep copies current. Apply processing technology directly where the data already lives. Example: running ML inference on an edge device or at a data center rather than moving petabytes of data.

Change Data Capture (CDC)

Detects and captures ONLY the incremental changes (inserts, updates, deletes) in source data, then moves those changes to the target system efficiently. Best for: high-velocity data where moving full data sets repeatedly is impractical. Minimizes data movement cost and latency. CDC is the correct choice when data changes FREQUENTLY and volume is HIGH. Moving all data repeatedly is inefficient — CDC moves only what changed.

Four Triggers for using Synthetic Data

1. Privacy & Security Constraints: Real-world data contains PII or sensitive information that would violate privacy laws (GDPR, HIPAA) or create security risks if used for training. Synthetic data provides a safe alternative without exposing real individuals.

2. Intellectual Property (IP) Constraints: Real data is proprietary or licensed in ways that prohibit its use for AI training. Synthetic data can replicate the statistical properties without the IP exposure.

3. Insufficient Real-World Data: The required data does not exist in sufficient quantity, variety, or completeness. Common in niche, emerging, or rare-event domains.

4. Class Imbalance (Rare Events): One class is severely underrepresented in training data (e.g., fraud cases = 2% of all transactions). Synthetic data generates additional examples of the minority class to balance training. Also known as oversampling.

Data Labeling Approaches:

Data labeling is the process of adding descriptive tags or metadata to training data to provide meaning for ML models — REQUIRED for supervised learning. Labels tell the model what the correct output should be for a given input.

• Manual Labeling: Humans use their knowledge to apply labels. Higher quality but slower and more expensive. Required when domain expertise is needed (e.g., medical imaging, legal documents).

• Automated Labeling: Systems infer labels based on previously trained models. Faster and cheaper but may introduce label errors. Requires validation to ensure quality.

• Bounding Boxes: Used in image labeling to identify and locate specific objects within an image. Machines see images as grids of pixels — bounding boxes tell the model which pixels constitute a meaningful object.

• Sensor Fusion: Correlates data from multiple sensor sources simultaneously. Example: autonomous vehicles combining lidar, radar, ultrasonic sensors, and cameras into a unified 3D point cloud model of the environment.

Representational Fairness vs. Technical Cleanliness

Technical data cleanliness (removing errors, duplicates, noise) is NOT the same as representational fairness (ensuring all relevant groups are adequately represented).

• Technical Cleanliness: Data is accurate, consistent, deduplicated, correctly formatted, and free of errors. A technically clean data set can still be biased. Addresses: Veracity, noise, format inconsistencies.

• Representational Fairness: All relevant groups, categories, demographics, and scenarios are adequately represented in the training data. A representationally fair data set may still have technical quality issues. Addresses: Who or what is included in the data — and in what proportions.

Why Both Matter: A model trained on technically clean but representationally unfair data will perform well on the majority group and poorly on underrepresented groups. CPMAI requires both technical cleanliness AND representational fairness for trustworthy AI. Both must be verified during Phase III — they are separate checks, not the same thing.

What is sensor fusion, and which AI pattern does it primarily support?

Correlating data from multiple sensor sources simultaneously — combining lidar, radar, ultrasonic, and camera inputs into a unified 3D point cloud. Primarily supports the Autonomous Systems AI pattern, where vehicles or robots must understand their full environment in real time.

Difference between a Data Warehouse and a Data Lake?

• Data Warehouse: stores data in a FIXED, structured schema — requires ETL (transform before loading).

• Data Lake: stores data in its NATIVE format (raw) — uses ELT (load first, transform later). Data lakes support big data environments with diverse, variable data formats

Phase 3 (Data Preparation) Go/No-Go

• All data preparation tasks have been actually executed (not just planned)

• Training, validation, and test data sets are prepared and split

• Data quality checks and verification are complete

• Data pipelines for both training AND inference are built and operational

• Labeling, augmentation, and enhancement requirements have been addressed

Three Types of Machine Learning?

• Supervised Learning: Trains on LABELED data to predict outputs. Human-annotated examples teach the model the correct answer. Tasks & CPMAI Pattern Alignment: Classification, Regression. Aligns with: Recognition, Predictive Analytics & Decision Support patterns.

• Unsupervised Learning: Finds patterns in UNLABELED data through discovery. No ground truth labels required. Tasks & CPMAI Pattern Alignment: Clustering, pattern discovery. Aligns with: Patterns & Anomalies pattern.

• Reinforcement Learning: Learns through trial and error in an interactive environment. Maximizes rewards over time. Tasks & CPMAI Pattern Alignment: Goal-driven optimization. Aligns with: Goal-Driven Systems pattern.

ML Algorithm vs. ML Model?

• ML Algorithm: The SET OF STEPS that tells the computer HOW to learn from data. Used during training. The method, not the result. Example: a neural network architecture specification.

• ML Model: The TRAINED OUTPUT produced when an algorithm runs on data. Used in PRODUCTION to generate predictions. What you are actually using when you say you are 'using an AI system.' Example: Chat GPT-4, a fraud detection classifier.

Key ML Algorithm Concepts

• Classification: Determines which category data belongs to. Binary (yes/no) or Multiclass (one of many). Examples: spam detection, image recognition. Uses supervised learning with labeled examples.

• Clustering: Automatically groups similar data without predefined categories. Unsupervised — no labels required. Example: customer segmentation. Finds hidden patterns humans might not discover manually.

• Regression: Predicts a continuous numerical value from input data. Discovers relationships between variables. Examples: home price prediction, demand forecasting, sales projections.

• Neural Network: A versatile architecture applicable to supervised AND unsupervised tasks. Interconnected layers: input, one or more hidden layers, output. More hidden layers = more sophisticated learning capability.

• Deep Learning: Neural networks with MORE THAN ONE hidden layer. 'Deep' refers to the number of hidden layers. Powerful and popular but computationally expensive and often NOT explainable (black box). If a simpler model achieves similar results, prefer the simpler option.

ML Model Taxonomy (Types)

• Pretrained Model: Already trained on a large relevant data set. Ideal when speed is critical, data is limited, or technical capability is constrained. Requires due diligence on provider — may have bias, privacy, or compliance risks from unknown training data. Readily available for Conversational and Recognition patterns; limited for Autonomous Systems.

• Foundation Model: Large pretrained model adaptable for a broad range of applications. Trained using SELF-SUPERVISED learning on massive data (billions of parameters, petabytes of data) — no labeled data required. Used directly for generic tasks OR fine-tuned for specific downstream tasks. Eliminates the need to train from scratch.

• Transformer Model: Type of deep learning neural network that transforms input sequences into output sequences. Uses encoder-decoder architecture with attention mechanism. Processes sequential data efficiently; tracks context within a context window. The basis for most modern LLMs.

• Large Language Model (LLM): Deep learning foundation models trained to generate human-understandable text or images from input prompts. Focus on natural language for input/output/training. Built on transformer architecture. PROS: ease of use, flexibility, fast results, wide knowledge base. CONS: not tailored to proprietary data, prone to hallucination, lack explainability (black box).

• Generative AI (GenAI): A type of ML that creates NEW data based on patterns learned from existing data. Text generators optimized for text; image generators for images. Applications: chatbots, synthetic data, code generation, content creation.

• AI Agent: Software that perceives its environment, makes decisions, and acts autonomously to achieve specific goals. Examples: virtual assistants, autonomous vehicles, recommendation systems.

• Agentic AI: Advanced AI where agents autonomously define, optimize, and iterate on workflows with minimal human involvement. Agents evaluate their own performance, redesign processes, and collaborate with other agents. Requires orchestration platforms, agent lifecycle management, and sophisticated monitoring.

Model Bias vs. Variance (Dartboard Framework)

• Bias: Degree predictions differ from target values. An ACCURACY problem. High bias = model makes faulty assumptions, misses patterns consistently. Causes UNDERFITTING (model too simple). Dartboard: arrows all miss in the same wrong direction.

• Underfitting (High Bias): Model too SIMPLE. Poor performance on BOTH training AND test data. Fix: use a more complex model.

• Variance: Degree model is sensitive to fluctuations in training data. A GENERALIZATION problem. High variance = model over-adapts to training data, fails on new data. Causes OVERFITTING (model too complex). Dartboard: arrows scattered widely around the target.

• Overfitting (High Variance): Model too COMPLEX. Excellent on training data, poor on new/unseen data. Fix: regularization, ensemble methods, or reduce model complexity.

• Well-Generalized Model: The 'sweet spot.' Captures meaningful patterns without over-adapting to training data. Performs consistently across training, validation, and test sets.

Overfitting vs. Underfitting

• Overfitting (High Variance): The model is too complex and has learned the training data too well — including its noise and irrelevant patterns. It performs excellently on training data but fails to generalize to new, unseen data. The model has essentially memorized the training set rather than learning the underlying patterns. Arrows are scattered widely — inconsistent, over-sensitive to small changes

• Underfitting (High Bias): The model is too simple and fails to capture meaningful patterns in the data. It performs poorly on both training data and new data because it has not learned enough from the training set. All arrows miss the target in the same wrong direction — consistently wrong

RAG vs. Fine-Tuning vs. Federated Learning

• Retrieval-Augmented Generation (RAG): A three-step process that grounds an LLM with specific data sources by: (1) retrieving relevant content from an external source, (2) augmenting the LLM prompt with that retrieved context, and (3) generating an output from the LLM that used the augmented prompt. No model retraining is required — RAG operates entirely at inference time. Handles frequently updated documents by refreshing the vector database rather than retraining

• Fine-Tuning: A technique that updates a pretrained model's weights by training it on domain-specific data to improve its performance on specialized tasks. Fine-tuning changes the model itself — it requires compute resources, training data, and ML engineering capability. Use fine-tuning when: Substantial domain-specific training data EXISTS, or The LLM does NOT provide accurate responses despite good prompt engineering.

• Federated Learning: A technique that distributes model training across multiple devices or servers WITHOUT sharing or centralizing the local data. The model goes to the data; the data never leaves its source. Each participating node trains on its local data and shares only the model updates (weights) — not the raw data — with a central aggregator. If training is happening but data cannot be centralized — that is Federated Learning.

Prompt Engineering - Key Patterns Reference

Prompt engineering is the practice of crafting input text to guide an LLM's output — avoids costly retraining for many tasks. The context window is the maximum input size an LLM can process, measured in tokens.

• Direct Asking: Simple, clear question or request. 'What is...', 'Can you...'. Best for: straightforward, unambiguous requests.

• Chain of Thought: Asks AI to reason step-by-step before answering. Best for: complex reasoning, math, logic problems where intermediate steps matter.

• Few-Shot Learning: Provides 2-3 input-output examples before the actual task. Best for: specific formatting requirements, creative writing in a particular style, consistent structured outputs.

• Role-Based Prompting: Asks AI to adopt a persona or expertise level. 'Act as a senior engineer.' Best for: tailoring responses to specific audiences, adjusting complexity.

• Template/Format Specification: Explicitly defines desired output structure. Best for: structured outputs like reports, code documentation, integration into existing workflows.

• Constraint-Based Prompting: Sets clear limits on response (word count, tone, inclusions/exclusions). Best for: content with specific requirements, professional communications.

• Iterative Refinement: Breaks complex tasks into steps, builds on prior responses. Best for: large projects, complex research, creative works needing development.

• Positive & Negative Examples: Shows both what you want AND what you do not want. Best for: tasks with subjective quality standards, when prior attempts missed expectations.

• Context-Rich Prompting: Provides comprehensive background and goals upfront. Best for: specialized domain tasks, when AI needs nuanced context for accurate responses.

Hyperparameter vs. Model Parameter

• Hyperparameter: Configuration settings determined by HUMANS BEFORE training begins. Controls how the model learns. Examples: learning rate, number of epochs, number of hidden layers, number of decision tree branches. Distinct from model parameters which are LEARNED BY THE MACHINE during training.

• Model Parameter: Values LEARNED BY THE MACHINE during training. Hyperparameters control how the model learns; parameters are what the model learned.

Hardware Types for Model Training (CPU, GPU, TPU)

• CPUs (Central Processing Units): General‑purpose processors designed to handle a wide variety of tasks sequentially. They are flexible and good for control logic, data preprocessing, and smaller workloads, but they are not optimized for large‑scale parallel computations required by deep learning.

• GPUs (Graphics Processing Units): Optimized for parallel mathematical computations, making them highly effective for training and running machine learning and deep learning models. Their ability to process many operations simultaneously made deep learning practical and widely adopted.

• TPUs (Tensor Processing Units): Specialized hardware accelerators designed specifically for machine learning workloads, particularly tensor operations used in deep learning. TPUs offer very high performance and efficiency for large‑scale model training and inference but are less flexible and typically tied to specific platforms or ecosystems.

What is Automated Machine Learning (AutoML)?

The tools, platforms, and processes that automate aspects of ML model building, including algorithm selection, hyperparameter tuning, feature engineering, model assessment. AutoML does NOT eliminate the need for quality training and test data. The data foundation (Phases I-III) is still essential regardless of AutoML usage.

What is Inference?

Inference is term used when you’re applying the model to new, real-world data​.

Phase 4 (Model Development) Go/No-Go Requirements

• You have built a model that meets the business requirements identified in Phase I

• You have adequate responses for all CPMAI Workbook Phase IV items — with well-defined answers to critical questions

• No unresolved roadblocks or warning signs exist that will prevent project success

Phase 4: When to Iterate Back to Phase 1, 2, & 3?

• Back to Phase III: data preparation issue (how data was prepared).

• Back to Phase II: data selection/understanding issue (which data was chosen).

• Back to Phase I: fundamental business problem, KPI feasibility, or Trustworthy AI/XAI constraint cannot be met. The root cause determines which phase to return to.

Phase 5 Model Evaluation Objectives (3)?

1. Tuning Model Performance — does the model properly fit training and test data?

2. Assessing Goal Performance — does it accomplish the business objectives and meet Phase I KPIs?

3. Assessing Technical Performance — does it meet technology KPIs (speed, cost, scalability)? All three must be satisfied before advancing to Phase VI.

False Positive vs. False Negative

• False Positive: The model flags something as TRUE when it is actually FALSE. Also called a Type I error. Example: spam filter marks a legitimate email as spam. Example: fraud model flags a legitimate transaction as fraudulent.

• False Negative: The model flags something as FALSE when it is actually TRUE. Also called a Type II error. Example: cancer screening model misses a true positive case. Example: fraud model fails to flag a genuinely fraudulent transaction.

Model Drift

The degradation in a MODEL'S PERFORMANCE over time as the real-world data and operational environment evolve from the conditions under which the model was originally trained. Models become 'stale' when their training data no longer reflects current reality. 3 types of Model Drift: Data Drift, Concept Drift, Source Drift

• Source Drift: New data sources with different characteristics from the original data sources the model was trained on. The new sources bring data patterns, distributions, and behaviors that were not present in the training data, causing the model's predictions to become less accurate over time. Something new is introduced (new channel, new campus, new customer segment, new data source). The model has never seen it before.

• Data Drift: The nature and quality of incoming data changes across existing sources

• Concept Drift: The way the model is used changes, or the relationship between inputs and outputs evolves

Data Drift

The gradual change in DATA CHARACTERISTICS over time, deviating from the original training distribution. The data the model sees in production no longer looks like the data it was trained on. Example: online vs. offline retail sales channel mix shifting after a supply chain disruption. 3 types of Data Drift: Feature Drift, Concept Drift, Label Drift

• Feature Drift: The DISTRIBUTION of input variables changes over time. The features themselves shift in their statistical properties. Example: the ratio of online vs. offline sales changes, altering the feature distribution the model learned from.

• Concept Drift: The RELATIONSHIP between input features and the target output variable changes. The model's predictions no longer map correctly to outcomes even if the features look similar. Example: user behavior evolves after a product redesign. Something existing gradually changes (fraud tactics evolve, user behavior shifts, clinical documentation practices change). The model has seen similar data but the relationship between inputs and outputs has drifted.

• Label Drift: The REPRESENTATIONS or categories used in the data become outdated or irrelevant. Example: classification labels from two years ago no longer reflect current product categories.

Techniques to Measure Data Drift?

• Histograms: Charts showing frequency distribution of data over time. Compare current data distributions against the original training distribution to identify shifts in feature values.

• Kolmogorov-Smirnov (KS) Statistic: A statistical measure that determines whether new data belongs to the SAME distribution as the training data. Used to detect FEATURE DRIFT If the KS statistic indicates significant distributional difference, retraining is likely needed.

• Target Distribution Monitoring: Monitor the distribution of model predictions and class labels over time. If the model's output distribution shifts significantly from the training distribution, it signals potential drift.

• False Positive/Negative Rate Monitoring: Regularly check false positive and false negative rates in production. Increasing rates in critical classes are a strong signal of model degradation requiring retraining.

Black Box vs. White Box Algorithms

• Black Box Algorithms: Produce opaque, difficult-to-explain results. The decision-making process cannot be easily traced or explained. Examples: deep learning neural networks, some ensemble methods. Powerful but lack explainability. NOT acceptable in regulated industries requiring decision justifications (finance, healthcare, government benefits).

• White Box (Glass Box) Algorithms Produce explainable, traceable results. The decision path can be followed step by step. Examples: linear models, decision-tree-based models. More explainable but typically do not achieve state-of-the-art performance. REQUIRED when explainability is a legal or regulatory obligation.

Three Triggers for Retraining a Model?

1. Performance Triggers: metrics fall outside acceptable ranges — distribution mismatch, unacceptable accuracy, class imbalance shifts.

2. Regular Retraining: proactive scheduled retraining even when performance is acceptable, based on known drift patterns.

3. Specific Scenarios: new data patterns, new sources, or significant operational changes. No universal fixed cadence — frequency depends on environment.

Systemic Transparency vs. Algorithmic Explainability (XAI)?

• Systemic Transparency: Comprehensive understanding of all COMPONENTS that went into creating the model: data sources, preprocessing steps, model architecture, training parameters. Tells you HOW the model was built. Does NOT explain why the model made a specific prediction.

• Algorithmic Explainability (XAI): The ability to understand and explain WHY the model made a specific prediction or decision. Required by law in many regulated contexts. Systemic transparency alone is NOT a substitute for algorithmic explainability when individual decision justification is required.

DevSecOps Integration for AI Monitoring

• Continuous Integration: AI is involved in continuous integration and feedback loops. Systems are continuously tested and integrated, catching issues early and making improvements in real time.

• Security and Compliance: Security protocols and compliance checks applied to AI systems as standard practice — not added as an afterthought. Same rigor as other parts of the technology stack.

• Automatic Documentation: Automated systems record and retain data from AI decisions and interactions. Reduces manual burden and minimizes human error in audit trail maintenance.

• Traceability Systems: Tools that provide a clear and detailed path from initial data input to final decision output. Enables transparency and accountability for users and regulators.

Phase 5 : Model Evaluation - Go/No-Go Requirements (3)

• The model meets the expected levels of fit based on its generalization performance

• The model meets both business AND technical KPIs established in Phase I

• A model iteration plan and monitoring approach are in place (drift detection, retraining triggers, pipeline for ongoing updates)

DO NOT ADVANCE IF The model fails any business or technical KPI. Bias issues have been identified but not resolved. Auditability and traceability requirements are not met. No plan exists for monitoring drift and retraining post-deployment.

Phase 5: Iteration Decision Matrix

• Iterate back to Phase IV when... Model performs poorly against KPIs — retrain, rebuild, or reconfigure. Hyperparameters need adjustment. Foundation model fine-tuning can be improved. Prompt engineering can be enhanced. Root cause: HOW the model was built.

• Iterate back to Phase III when... Model performance issues are related to data QUALITY — better labeling, augmentation, or a different data subset could resolve them. Root cause: HOW the data was prepared.

• Iterate back to Phase II when... Data is fundamentally flawed or inadequate. Training data does not match real-world data. Need different or alternate data sources. Root cause: WHICH data was selected.

• Iterate back to Phase I when... Model consistently fails KPIs despite all other remediation. Fundamental issue with the business problem itself. No fine-tuning or better data will resolve the core gap. Root cause: THE BUSINESS PROBLEM needs rethinking.

Phase 6: Model Operationalization - Four Key Activities

1. Determine Deployment Location: Decide where and how the model will be used: on-premise, cloud, hybrid, edge devices, or API-based access. The deployment environment must match the model's resource requirements established in Phase V evaluation.

2. Continuous Monitoring: Monitor model performance while in use — performance will change over time due to data drift, model drift, and source drift. Monitoring is not optional; it is a mandatory Phase VI activity. Dashboards, alerts, and automated monitoring tools should be in place at deployment.

3. Model Optimization: Ongoing tuning and improvement of the deployed model based on real production data and user feedback. Different from model development — optimization in Phase VI uses live operational data, not training data.

4. Model Governance: Implementing and enforcing the governance framework: audit trails, access controls, compliance checks, bias monitoring, version control, and human oversight processes. Governance must be continuous, not a one-time checkpoint.

Issues discovered in Phase VI (performance degradation, new bias findings, changed business requirements) trigger iteration back to the appropriate earlier phase. The CPMAI methodology is circular, not linear.

AI Operational Frameworks (3)

1. MLOps (Machine Learning Operations): Practices and tools for managing the LIFE CYCLE of machine learning models from development to deployment and monitoring. Draws on DevOps principles applied to ML. Enables reliable, repeatable, and automated model deployment, monitoring, versioning, and retraining. CPMAI cites MLOps as a key tool for Phase V and Phase VI monitoring and management.

2. ModelOps: The broader practice of operationalizing and managing ALL types of AI and analytical models in production — including ML models, rules-based systems, and optimization models. Extends MLOps to cover the full range of model types an organization may deploy.

3. DataOps: A set of practices and technologies for managing the data life cycle in an agile, automated manner to improve quality and speed. Addresses the continuous data pipeline needs of AI systems in production — including drift detection, retraining data pipelines, and data quality monitoring.

Five-Layer Trustworthy AI Framework

1. Ethical AI: What AI systems SHOULD OR SHOULD NOT DO to be ethical according to society. Guidelines for moral conduct in AI development. Values: dignity, fairness, diversity and inclusion, bias mitigation, freedom and human agency, human oversight, sustainability. Addresses: safety threats, fraud and misuse, inadequate practices, privacy. Societal layer. Foundational. Addresses the broadest human values questions. Ethics is about values.

2. Responsible AI: Focuses on LAWS AND REGULATIONS governing AI use. Ensures compliance with legal frameworks at local, national, and international levels. Addresses how to prevent misuse and abuse. Requires knowing and implementing applicable regulatory requirements. Systemic-ethical boundary layer. Responsibility is about legal obligations.

3. Transparent AI: Provides VISIBILITY INTO AI SYSTEMS — how they were built, what data they use, how biases are measured and mitigated, and what decisions they make. Two levels: (1) Algorithmic explainability (how individual decisions are made) and (2) Systemic transparency (visibility into all components, data, and configurations). Includes disclosure and consent when AI is used. Systemic layer. Includes both technical and process transparency.

4. Governed AI: The PRACTICES AND PROCESSES put in place to ensure AI operates ethically, responsibly, and transparently. Includes: audits, measurements, regulations, bias mitigation, system audits, contesting results, continuous quality checks, external regulations, third-party certifications, internal training. Governance ensures commitments to trustworthy AI are followed, not just stated. Systemic layer. Where policy meets operational practice.

5. Interpretable and Explainable AI: Making AI ALGORITHMS EXPLAINABLE AND UNDERSTANDABLE at the technical level. Focuses on algorithmic explainability, algorithmic interpretability, and root cause explanations. Aims to eliminate black boxes where possible. If a more explainable algorithm can achieve acceptable results, it should be selected over a more opaque one. Technical layer. The most specific layer — explains individual algorithm decisions.

The five-layer framework is a set of GUIDELINES, not a strict process that must be followed in layer order for every project. Organizations select what fits their specific project needs. There is no CPMAI mandate that Layer 1 must be fully resolved before Layer 2 can begin. Ethics guides the spirit; regulations define the legal boundaries of what is permissible.

Four Elements of Systemic Transparency (Transparent AI)

1. System Transparency: AI systems must reveal the data and system components utilized in producing outcomes, along with their configurations. Human choices regarding management, updates, creation, and use of AI systems must be disclosed and made accessible.

2. Bias Measurement and Mitigation: AI systems must provide a way to constantly measure bias from various sources and mitigate detected bias. Without measurement: you cannot know if you are serving your audience effectively, cannot avoid legal liability, cannot fix issues, and cannot prove trustworthiness.

3. Open Systems: AI systems should ideally use open-source technology with a mechanism that enables anyone to view how the systems operate. Promotes accountability and independent verification.

4. Disclosure and Consent: Organizations must disclose when AI systems are being used and when humans are interacting with AI rather than a human. Users must be given the option to opt out of AI interactions. Active consent is required.

Why AI projects fail - 3 Root Causes

1. Treated as Application Projects: The root cause of most AI failures. AI systems learn from data; they are NOT traditional software. Application projects specify and control functionality. Data projects extract insights from data regardless of its current form. Teams focused on features and functionality miss the data-centric nature of AI. CPMAI exists to prevent this.

2. Data-Related Issues: Continuously changing data (quality and consistency require ongoing management). Data governance and security gaps (access control, provenance, privacy). Data ownership conflicts (different groups own data, sharing is challenging). Data quality management failures. Data consistency issues across sources. Distributed functionality challenges.

3. Unable to Address AI-Specific Issues: Proof-of-concept trap: building PoCs on ideal controlled data that fail in real-world conditions. Insufficient data quality and quantity. AI system not updated with evolving real-world data. Other AI-specific failure modes addressed throughout the CPMAI phases.

The Proof of Concept (POC) Trap vs. The Pilot Approach

46% of organizations discarded AI proofs of concept before they reached production. CPMAI advocates going directly to a minimum viable product (MVP) pilot.

• Proof of Concept (PoC) — AVOID: Performed in a controlled environment with limited, idealized data. Demonstrates that something CAN be built, not that it WORKS in production. High failure rate and low adoption rate when scaled to real-world scenarios. Real-world data is messier and more complex than PoC data. Most effort invested may be thrown away.

• Pilot (Minimum Viable Product) — RECOMMENDED: Uses REAL-WORLD DATA and ACTUAL CUSTOMERS in a safe, controlled environment. Delivers actual value from the start. Identifies and solves problems early with real data and real users. Tests with users unfamiliar with the AI system to gauge real-world usability. Allows iterative improvement toward production rather than starting over. In lean agile terms: the minimum viable product (MVP).

Emotional Fears vs. Rational Concerns about AI

• Emotional Fears (Hard to Address with Logic): Will AI take over the world? Will robots take away jobs? Will privacy and data be lost? Will data and power concentrate in few hands? These are emotional responses that may be resistant to factual counterarguments. CPMAI requires awareness and empathy in stakeholder management.

• Rational Concerns (Addressable with Technical/Governance Measures): Bias and algorithmic discrimination. Data privacy and misuse. Insufficient laws and regulations. Lack of transparency in AI decisions. Job displacement through automation. These concerns can be addressed through the trustworthy AI framework, governance practices, explainability measures, and human oversight.

CPMAI vs. CRISP-DM vs. Agile

• CRISP-DM (Cross-Industry Standard Process for Data Mining): Released 1999. Popular structured method for data mining projects covering business understanding, data preparation, modeling, evaluation, and deployment. DATA-CENTRIC but not optimized for AI/ML specifically. Not widely adopted outside data mining. Not modified for iterative or agile AI development. CPMAI extends CRISP-DM with AI and ML-specific documents, processes, and tasks.

• Agile: Widely adopted iterative development approach. Breaks projects into sprints, emphasizes continuous feedback and adaptive planning. NOT data-centric. Does not address how to manage data-specific iterations in AI projects. Does not account for AI's unique failure modes, data dependencies, or trustworthy AI requirements.

• CPMAI (Cognitive Project Management in AI): Vendor-neutral methodology for AI, ML, advanced data analytics, intelligent automation, and cognitive projects of any size. Builds on CRISP-DM and updates agile with data science and data management methodologies. Uses the seven patterns of AI and the trustworthy AI framework for guidance. All steps are iterative. Data-first, AI-relevant, highly iterative, focused on operational success. Best of both worlds: CRISP-DM's data focus + agile's iteration + AI-specific requirements.

Complete Phase Iteration Decision Matrix: Use this matrix to determine where to iterate back to.

• Phase I — Business Understanding: Business problem is undefined or infeasible. KPIs cannot be achieved. Trustworthy AI/XAI requirements cannot be met. Models consistently fail KPIs despite all other remediation.

• Phase II — Data Understanding: Training data does not match real-world operational data. Model performance is poor due to data SELECTION. Pretrained/foundation model does not match available data. Data does not exist or cannot be obtained.

• Phase III — Data Preparation: Prepared data is not suitable for the model. Data labeling is inadequate. GenAI responds poorly to prepared prompts. Fine-tuned models fail with prepared data. Data pipeline is technically/legally infeasible.

• Phase IV — Model Development: Model performs poorly against KPIs but data and business problem are sound. Hyperparameters need adjustment. Fine-tuning or prompt engineering can be improved.

• Phase V — Model Evaluation: Model evaluation reveals performance, bias, or compliance gaps that prevent deployment. KPIs not met. Auditability/traceability incomplete. No drift monitoring plan.

• Phase VI — Triggers re-entry to I, II, III, IV, or V depending on root cause: Post-deployment performance degrades, new regulatory requirements emerge, or business needs change. These trigger re-evaluation and return to appropriate earlier phase.

Ask these questions IN ORDER: (1) Is the business problem definition or a legal constraint the issue? → Phase I. (2) Is wrong data selected or unavailable? → Phase II. (3) Is data preparation quality the issue? → Phase III. (4) Is model building the issue? → Phase IV. (5) Is the model not ready for deployment? → Phase V. Stop at the FIRST yes.

Evaluation Metrics

• Accuracy: The overall percentage of correct predictions across all classes. Calculated as correct predictions divided by total predictions. Useful when classes are balanced. Misleading when data is imbalanced — a model predicting only the majority class achieves high accuracy trivially without learning anything useful.

• Precision: Of all the cases the model flagged as positive, what percentage were actually positive? High precision means few false positives. Use precision as the primary metric when false positives are costly — for example, a spam filter incorrectly marking legitimate emails, or a fraud model flagging legitimate transactions and frustrating customers.

• Recall (also called Sensitivity): Of all actual positive cases in the dataset, what percentage did the model correctly identify? High recall means few false negatives. Use recall as the primary metric when missing a true positive is the most harmful error — cancer screening missing a real diagnosis, a fraud model missing a genuine fraud case, or a churn model missing a customer who will actually leave.

• F1 Score: The harmonic mean of precision and recall. Balances both metrics into a single number. Use F1 when you need one metric that accounts for both false positives and false negatives simultaneously. The go-to metric for imbalanced datasets where overall accuracy is misleading.

• Precision-Recall AUC (Area Under the Precision-Recall Curve): Measures model performance specifically on the minority class across all classification thresholds. Used instead of ROC-AUC when data is highly imbalanced. Better captures how well the model identifies the rare, important class. Cited in the telecom churn and insurance fraud case studies.

• ROC-AUC (Receiver Operating Characteristic — Area Under the Curve): Measures the model's discriminative power across all classification thresholds — its ability to distinguish between positive and negative classes. A general-purpose metric for balanced classification problems. The IT Help Desk chatbot example targets an AUC of approximately 0.88

Data Stewardship vs. Data Quality Management

• Data Stewardship: Enforces governance

• Data Quality Management: Improves data accuracy

Proof Of Concept (PoC) Trap Statements

True:

• 46% of AI PoCs are discarded before production

• CPMAI recommends going directly to a pilot with real-world data and real users instead

False:

• CPMAI recommends proofs of concept as the preferred starting point for all AI projects because they minimize early investment risk

• A proof of concept is equivalent to a minimum viable product in CPMAI's framework — both use real-world data from the start

• Proofs of concept are acceptable when the project uses a foundation model because foundation models are not sensitive to idealized training data