Data Mining (Prelim Module 1) Reviewer

Module 1

Machine Learning (ML)

A branch of Artificial Intelligence (AI). It has become an extensively used tool for websites to classify users and address them appropriately.

Example of tasks difficult for rule-based programming

Photo tagging, spam classification, web page ranking.

Training data

Examples provided to a program so it can generate its own rules.

Arthur Samuel in 1959.

Origin of the term "machine learning"

Arthur Samuel’s definition of Machine Learning

"The field of study that gives computers the ability to learn without being explicitly programmed."

Tom Mitchell’s formal definition of Machine Learning (1998)

"A computer program is said to learn from experience E with respect to some class of tasks T and performance measures P, if its performance at tasks in T, as measured by P, improves with experience E."

Virtual Personal Assistants, Traffic Predictions, Online Transportation Networks, Video Surveillance, Social Media Services, Email Spam and Malware Filtering, Online Customer Support, Search Engine Result Refining, Product Recommendations, Online Fraud Detection, Medicine, Computational Biology, Handwriting Recognition, Machine Translation, Driverless Cars and Autonomous Helicopters.

Applications of Machine Learning

Virtual Personal Assistants

Assistants like Siri, Alexa, and Google Assistant use ML to discover information via voice commands, answer queries, and perform tasks like setting alarms or reminders.

Smart speakers as ML products

Amazon Echo and Google Home are products of ML-driven virtual personal assistants.

Traffic Predictions using ML

Google Maps utilizes ML to predict estimated arrival times and model real-time traffic congestion.

Online Transportation Networks and ML

Apps like Ola and Uber employ ML to estimate ride prices and predict rider demand for surge pricing.

ML in Video Surveillance

ML-based systems help in monitoring CCTV cameras by training computers to detect unusual behavior and alert human attendants.

Social Media Services and ML personalization

ML personalizes news feeds for better advertisement targeting.

Email Spam Filtering with ML

Identifies spam based on previously marked emails.

Malware detection using ML

Security programs detect new malware by understanding coding patterns, even when malware code is 90-98% similar to older versions.

Online Customer Support with ML chatbots

Extracts information and provides responses to customer queries without live officials.

Search Engine Result Refining with ML

Improves search results by observing user responses to displayed results.

E-commerce Product Recommendations with ML

Amazon and Flipkart recommend products based on user behavior, past purchases, liked items, cart additions, and brand preferences.

Online Fraud Detection with ML

PayPal compares millions of transactions to distinguish between legitimate and illegitimate ones.

ML in Medicine

Algorithms help doctors understand diseases better by transforming electronic medical records into medical knowledge.

ML in Computational Biology

Assists in understanding and identifying relationships between genes and human features from vast human DNA data.

ML in Handwriting Recognition

Recognizes and reads different handwritings, useful in postal mail routing.

Machine Translation with ML

Technologies like Google Translate instantly translate text between numerous human languages.

ML in Driverless Cars

Crucial for self-driving cars where defining explicit rules is challenging.

ML in Autonomous Helicopters

Enables autonomous helicopters to learn through experience.

Supervised Learning definition

The algorithm is provided with labeled data (inputs with corresponding outputs) to learn and predict unseen data.

Regression problems definition

Predict results within a continuous output.

Examples of regression problems

Predicting student marks, cricket match scores, tomorrow's temperature.

Classification problems definition

Predict results in a discrete output.

Examples of classification problems

Predicting student grades/divisions, whether a cricket team will win or lose, or if tomorrow will be cooler or hotter.

Unsupervised Learning definition

Deals with problems with no labeled data, aiming to derive structure via grouping or clustering.

Google News

Clusters news items based on content attributes like word frequency, sentence length, and page count.

Social Network Analysis

Clusters users into groups based on interests, connections, and communication patterns.

Market Segmentation

Groups customers based on purchase history to target promotions.

Astronomical Data Analysis

Uses light patterns to identify galaxies, planets, and satellites.

Reinforcement Learning definition

Works through trial and error to maximize specified rewards over time.

Key components of reinforcement learning

Agent, Environment, Actions.

Agent in reinforcement learning

Responsible for learning and decision-making.

Environment in reinforcement learning

External world with which the agent interacts.

Actions in reinforcement learning

Tasks performed by the agent.

Data Mining

is the result of the natural evolution of information technology, specifically through the utilization of data management systems.

Data Mining

is the process of digging through a collection of data to discover hidden connections and patterns for the prediction of future trends.

Data mining

Also referred to as knowledge discovery from data.
Data Mining as Interdisciplinary Subfield

Predictive analytics

In business analytics, data mining is referred to as?

Applications of Data Mining: Target Marketing

Data mining is used to understand customers better and segment market groups for customized promotional campaigns.

Applications of Data Mining: Credit Risk Management

Used by financial institutions to predict a borrower’s ability to repay debt and assign interest rates based on risk.

Applications of Data Mining: Fraud Detection

Applied to detect and stop fraudulent transactions automatically by tracking spending habits.

Applications of Data Mining: Healthcare

Predicts a patient’s likelihood of different health conditions based on risk factors like demographic, family, and genetic data.

Applications of Data Mining: Sentiment Analysis

Gathers understanding of how a group feels towards a topic, often using social media data.

Applications of Data Mining: Recommender Systems

Recommends products and services based on consumer behavior.

Applications of Data Mining: Spam Filtering

Analyzes characteristics of malicious messages.

Applications of Data Mining: Education

Predicts students’ future learning behavior and analyzes outcomes for teaching strategy improvement.

Applications of Data Mining: Criminal Investigation

Detects crimes and their relationships with criminals.

Benefits of Data Mining: Automated Decision-Making

Enables continual analysis and automation of routine and critical decisions without human judgment delay.

Benefits of Data Mining: Accurate Prediction and Forecasting

Provides reliable forecasts based on past trends and current conditions.

Benefits of Data Mining: Cost Reduction

Allows efficient resource allocation for maximum cost reduction.

Benefits of Data Mining: Customer Insights

Creates customer personas and personalizes touchpoints to improve experience.

Data Quality

Challenges of Data Mining: Accuracy, completeness, and consistency of data affect results.

Causes of data quality issues

Data entry errors, storage issues, integration problems, transmission errors.

Methods to address data quality issues

Data cleaning (detecting/correcting errors) and data preprocessing (transforming data for mining).

Data Complexity

Challenges of Data Mining: Large, varied data from sources like sensors, social media, IoT, often in different formats.

Techniques for handling data complexity

Clustering, classification, association rule mining.

Data Privacy and Security

Challenges of Data Mining: Increased risk of breaches and cyber-attacks with more data collection.

Scalability

Challenges of Data Mining: Algorithms must handle large datasets and streaming data efficiently.

Interpretability

Challenges of Data Mining: Complex models can be difficult to understand.

Ethics

Challenges of Data Mining: Concerns about discrimination, privacy violation, perpetuating bias.

Volume

Six Vs of Data: Refers to the huge amount of data; size determines if it qualifies as Big Data.

Example of projected Volume in 2020

Almost 40,000 Exabytes of data.

Velocity

Six Vs of Data: High speed of data accumulation from sources like machines, networks, social media, mobile phones.

Variety

Six Vs of Data: Refers to structured, semi-structured, and unstructured data from heterogeneous sources.

Structured data

Organized data with defined length and format.

Semi-structured data

Semi-organized data not conforming to formal structures; example: log files.

Unstructured data

Unorganized data not fitting into relational database tables; examples: text, pictures, videos.

Veracity

Six Vs of Data: Inconsistencies and uncertainty in data quality and accuracy.

Value

Six Vs of Data: Most important “V”; bulk data is useless unless turned into something valuable.

Variability

Six Vs of Data: Refers to changes in data structure or meaning over time.

KDD (Knowledge Discovery in Databases)

Process of extracting useful, unknown, and valuable information from large datasets.

KDD Step 1: Data Selection

Identifying and gathering relevant data from available sources, filtering out irrelevant data or noise.

KDD Step 2: Data Cleaning

Improving data quality by correcting errors, handling missing values, removing outliers, fixing duplicates.

KDD Step 3: Data Integration

Combining data from multiple sources into a unified dataset, resolving mismatches and redundancies.

KDD Step 4: Data Selection and Transformation

Selecting relevant data and preparing it by converting into suitable formats.

KDD Step 5: Data Mining

Applying algorithms like classification, clustering, association rule mining, regression, anomaly detection.

KDD Step 6: Pattern Evaluation

Assessing discovered patterns for relevance and usefulness.

KDD Step 7: Knowledge Representation

Presenting knowledge in charts, reports, dashboards for decision-making.

Improves decision-making

Advantages of KDD: Provides valuable insights to help organizations make better decisions.

Increased efficiency

Advantages of KDD: Automates repetitive tasks and prepares data for analysis.

Better customer service

Advantages of KDD: Helps understand customer needs and preferences.

Fraud detection

Advantages of KDD: Identifies patterns or anomalies indicating fraud.

Predictive modeling

Advantages of KDD:Builds models to forecast trends.

Privacy concerns

Disadvantages of KDD: May involve sensitive personal information.

Complexity

Disadvantages of KDD: Requires specialized skills and knowledge.

Unintended consequences

Disadvantages of KDD: May perpetuate bias or discrimination.

Data Quality

Disadvantages of KDD: Relies heavily on quality; inaccurate data misleads results.

High cost

Disadvantages of KDD: Requires large investments in resources.

Overfitting

Disadvantages of KDD: Model learns noise in training data, reducing performance on new data.