DSS110 | Lesson 1 - 2

Introduction to Data Science

• Is a interdisciplinary field

• Uses scientific method, processes, algorithms and systems to extract knowledge and insights from structured and unstructured data

What is Data Science

• Domain Experience

• Mathematics & Statistics

• Computing Skills

3 Main domains of Data Science

• Helps organization understand customers.

• Improves processes and operations.

• Enables innovation and new products.

• Creates business value and competitive advantage.

Why is Data Science Important? ( can read )

• Data

• Information

• Knowledge

• Insight

• Decision

5 Steps from Data to Decisions

Raw facts and figures

What is Data

Organized and meaningful data

What is Information

Understanding derived from information

What is Knowledge

Patterns that lead to actioin

What is Insight

Informed choices for better results

What is Decision

Statistical Analysis

• Hypothesis testing

• Traditional statistics

• 1960s - 70s

Data Mining

• Discover patterns in data

• Focus on algorithms

• 1980s - 90s

Big Data

• Large volume

• Variety

• Velocity

• Veracity

• 2000s - 2010s

Data Science

• Integrates statistics, ML, computing, domain knowledge

• End-to-end value creation

• 2010s - present

• Volume

• Velocity

• Variety

• Veracity

• Value

What are the 5 V’s of Big Data

Distributed Storage

What Tool / Data Infrastructure are

• Hadoop

• HDFS

• Cloud Storage

from

Processing Engines

What Tool / Data Infrastructure are

• MapReduce

• Spark

from

NoSQL Databases

What Tool / Data Infrastructure are

• MongoDB

• Cassandra

• HBase

from

Data Ingestion

What Tool / Data Infrastructure are

• Kafka

• Flume

from

Cloud Platforms

What Tool / Data Infrastructure are

• AWS

• Azure

• GCP

from

• Machines that can perform tasks that typically require human intelligence

What is Artificial Intelligence

• Subset of AI that allows machines to learn from data.

What is Machine Learning

• Subset of ML using neural networks with many layers.

What is Deep Learning

• Problem Understanding

• Data Collection

• Data Preparation

• Exploratory Data Analysis (EDA)

• Modeling

• Evaluation

• Deployment

• Communication

8 Steps of Data Science Lifecycle

Problem Understanding

• Define the problem clearly

• Identify goals and objectives

• Determine success metrics

• Understand constraints

• Translate to Data Science problem

Data Collection

• Identify data sources

• Collect relevant data

• Ensure data quality

• Store data securely

• Types :

  • Structured

  • Unstructured

  • Semi-Structured

Data Preparation

• Clean missing values

• Handle outliners

• Transform and normalize

• Integrate datasets

• Feature engineering

Exploratory Data Analysis (EDA)

• Understand data summaries and statistics

• Visualize distributions

• Find patterns and relationships

• Generate hypothesis

Modeling

• Select appropriate algorithms

• Train models on data

• Tune hyperparameters

• Uses cross-validation

Evaluation

• Evaluate using metrics

• Compare models

• Avoid overfitting

Deployment

• Deploy model to production

• Integrate with applications

• Monitor performance

• Handle data drift

Communication

• Tell the story with data

• Visualize key insights

• Provide actionable recommendations

• Report to stakeholders