Data Science Prelim

Data Analytics

It is a combination of processes to extract information from datasets

Data Analytics Process

Business Problem > Data Acquisition > Data Wrangling > Exploratory Data Analysis (EDA) > Data Exploration > Conclusion or prediction > Communication

Data acquisitions

Its goals are to collect data from various data sources, web server logs, and scrape through database

Data wrangling

It is the most challenging phase and takes up 70% of the the Data Scientist's time.

Exploratory Data Analysis’ four section

Approach, Focus, Assumption, and EDA Techniques: Quantitative or Graphical

EDA’s Approach

EDA approach studies the data to recommend suitable models that fit the data.

EDA’s Focus

The focus is on data; its structure, outliers, and models suggested by the data.

EDA’s Assumption

EDA techniques make minimal or no assumptions. They present and show all the underlying data without any data loss.

EDA Techniques: Quantitative Technique

Provides numeric outputs for the inputted data.

EDA Techniques: Graphical Technique

Uses statistical functions for graphical output.

Histograms and Scatterplots

The two popular graphical techniques to depict data.

Histograms

Graphically summarize the distribution of a univariable dataset.

Scatter plots

Represents relationship between two variables

Conclusion or Prediction

This step involves reaching a conclusion and making predictions based on the data analysis.

Hypothesis

It is used to establish the relationship between dependent and independent variables.

Hypothesis Testing Process

Choosing the training and test dataset, and evaluating term with the null and alternative hypothesis.

Null Hypothesis

Proposed model does not predict better than the existing model

Alternative Hypothesis

Proposed model predicts better than the existing model

Communication

The last step of the data analysis is communication, where the analyzed data is formally presented to stakeholders

Data Visualization

Techniques are used for effective communication of data.

Forms of Data analysis presentations:

* Visual Graphs
* Plotting maps
* Reports
* Whitepaper reports
* PowerPoint Presentations

Plotting

is a data visualization technique used to represent underlying data through graphics.

plotting technique is used for continuous data

Line chart and Histograms

The main machine learning library for Python

Scikit-learn

Data Wrangling

Includes data transformation, merging, aggregation, group by operation and reshaping

Median

Measure of central tendency is used to catch outlier in the data

Training dataset

The proposed model is built on the training dataset in hypothesis testing.

Beautiful soup library

Used for Web Scraping, mainly used in the data acquisition

Data Scientist’s roles

* Ask the right questions
* Understand data structure
* Interpret and wrangle data
* Apply statistical and mathematical methods
* Visualize data and communicate with stakeholders
* Work as a team player

**3 Vs of Big Data**

Volume, Velocity, and Variety

Principles of Computational Thinking

Decomposition, Abstraction, Pattern Recognition, and Algorithm Design

Decomposition

This is the breaking down a complex problem or \n system into smaller, more easily solved parts.

Abstraction

Identifying the general principles that generate these \n patterns. Allows us to create a general idea of what the \n problem is and how to solve it

Pattern Recognition

Observing patterns, trends, and regularities in \n data.

Algorithm Design

Developing the step by step instructions for solving this and similar problems.

Computational Thinking

A problem solving process that includes a number of characteristics and dispositions.

Descriptive

Study a dataset to decipher the details

Predictive

Create a model based on existing information to predict outcome and behavior

Prescriptive

Suggest actions for a given situation using the collected information