DSS110 (Lesson 1 -4)

Data science

This is one of the fastest-growing fields in the world.

The demand for this career is expected to increase in the future.

Data science

______ is an interdisciplinary field that uses scientific methods, processes, algorithms and systems to extract knowledge and insights from structured and unstructured data.

Data science

________ is the adaptive, iterative, and phased approach to the analysis of data, performed within a systematic framework, that uncovers optimal models by assessing and accounting for the true costs of prediction errors.

Data science

This field combines statistical data analysis, computer science, and business intelligence to uncover nuggets of information from large databases.

Data science

This multidisciplinary field allows us to extract knowledge from databases.

Data

This is what drives decisions. It is characterized as raw facts and figures.

Information

This is when data becomes organized and meaningful.

knowledge

This is the understanding that is derived from information.

insight

These are patterns that lead to action.

decision

These are informed choices for better results.

Data mining

The era that immediately preceded Big Data (2000s-2010s) and focused on discovering patterns through algorithms was called __________.

Big data

The __________ era, spanning the 2000s–2010s, is characterized by large volume, variety, velocity, and veracity.

Statistical analysis

Traditional statistics and hypothesis testing were the main focus areas during the __________ era (1960s–70s).

Data science

The __________ era is known for achieving end-to-end value creation.

• volume

• velocity

• variety

• veracity

• value

Big Data is characterized by the 5V’s. What are the 5V’s?

Big data

NoSQL databases, cloud storage, Azure, and GCP are examples of __________ infrastructure and tools.

artificial intelligence

This is characterized by machines capable of performing tasks that typically require human intelligence.

Machine learning

This is a subset of AI that allows machines to learn from data

Deep learning

This is a subset of machine learning that uses neural networks with many layers

Data analytics lifecycle

The __________ is designed specifically for Big Data problems and data science projects.

six

The Data analytics lifecycle has ____ phases, and project work can occur in several phases at once.

business user

This key role in an analytics project is characterized by someone who understands the domain area and usually benefits from the results.

business user

This key role in an analytics project is characterized by someone who can consult and advise the project team on the context of the project, the value of the results, and how the outputs will be operationalized.

project sponsor

This key role in an analytics project is characterized by someone who is responsible for the genesis of the project.

project sponsor

This key role in an analytics project is characterized by someone who provides the impetus and requirements for the project and defines the core business problem.

project manager

This key role in an analytics project is characterized by someone who ensures that key milestones and objectives are met on time and at the expected quality.

business intelligence analyst

This key role in an analytics project is characterized by someone who provides business domain expertise based on a deep understanding of the data, KPIs, key metrics and business intelligence from a reporting perspective.

business intelligence analyst

This key role in an analytics project is characterized by someone who generally creates dashboards and reports and knows the data feeds and sources.

database administrator

This key role in an analytics project is characterized by someone who provisions and configures the databases to be used by the working team.

data engineer

This key role in an analytics project is characterized by someone who executes the actual data extractions and performs substantial data manipulation to facilitate analytics.

data scientist

This key role in an analytics project is characterized by someone who designs and executes analytical methods and approaches with the data available to the project.

phase 1

In this phase, the team learns the business domain, including relevant history, such as whether the organization or business unit has attempted similar projects in the past from which they can learn.

phase 2

In this phase, the presence of an analytic sandbox is required, in which the team can work with data and perform analytics for the duration of the project.

ETL, ETL

In the phase 2, data preparation, the team needs to execute ____ or ____ to get data into the analytic sandbox.

extract, transform, and load

What does ETL stand for?

execute, load, and transform

What does ELT stand for?

ETLT

The ETL and ELT are sometimes abbreviated as _______.

phase 2

In this phase, the team also needs to familiarize itself with the data thoroughly and take steps to condition the data.

phase 3

In this phase, the team determines the methods, techniques, and workflow it intends to follow for the subsequent model building phase,

phase 3

In this phase, the team explores the data to learn about the relationships between variables and subsequently selects key variables and the most suitable models.

phase 4

In this phase, the team develops datasets for testing, training, and production purposes.

phase 4

In this phase, the team builds and executes models based on the work done in the model planning phase.

phase 4

In this phase, the team also considers whether its existing tools will suffice for running the models, or if it will need a more robust environment for executing models and workflows.

phase 5

In this phase, the team, in collaboration with major stakeholders, determines if the results of the project are success or a failure based on the criteria developed in phase 1.

phase 5

In this phase, the team should identify key findings, quantify the business value, and develop a narrative to summarize and convey findings to stakeholders.

phase 6

In this phase, the team delivers final reports, briefings, code, and technical documents. In addition, the team may run a pilot project to implement the models in a production environment.

phase 1

In this phase, the team formulates initial hypotheses that can later be tested with data

phase 1

In this phase, the team needs to determine how much business or domain knowledge the data scientist needs to develop models in phases 3 and 4.

phase 1

In this phase, the team needs to assess the resources that are available to support the project. These resources include technology, tools, systems, data, and people.

framing

This is the process of stating the analytics problem to be solved.

phase 1

In this phase, it is important to establish a failure criteria.

failure criteria

The _______ that was established in Phase 1 will help guide the team in understanding when it is best to stop trying or settle for the results that have been gleaned from the data.

phase 1

In this phase, it is also important to identify the key stakeholders and their interests in the project. The team should plan to collaborate with the stakeholders to clarify and frame the analytics problem.

phase 2

This phase includes the steps to explore, preprocess, and condition data prior to modeling and analysis.

phase 2

In this phase, the team needs to create a robust environment in which it can explore the data that is separate from a production environment. This is done by preparing an analytics sandbox.

phase 2

In this phase, the team also must decide how to condition and transform data to get it into a format to facilitate subsequent analysis.

Data preparation

__________ tends to be the most labor-intensive step in the analytics lifecycle.

phase 2

This phase is generally the most iterative and the one that teams tend to underestimate most often.

workspace

An analytic sandbox is also commonly referred to as a _________, in which the team can explore the data without interfering with live production databases.

load, transform

The analytic sandbox approach differs slightly as it advocates extract, ______, and then ______.

In this phase, it is important to catalog the data sources that the team has access to and identify additional data sources that the team can leverage.

Data conditioning

This refers to the process of cleaning data, normalizing datasets, and performing transformations on the data.

Data visualization

This is utilized to examine data quality, such as whether the data contains many unexpected values or other indicators of dirty data.

Ben Shneiderman’s mantra

This is a pragmatic approach to visual data analysis.

It enables the user to find areas of interest, zoom in and filter to find more detailed information about a particular area of the data, and then find the detailed data behind that area.

Ben Shneiderman’s mantra

This approach provides a high-level view of the data and a great deal of information about a given dataset in a relatively short period of time.

phase 3

In this phase, the team identifies candidate models to apply to the data for clustering, classifying, or finding relationships in the data depending on the goal of the project.

phase 3

In this phase, the structure of the datasets is assessed to dictate the candidate tools and algorithms.

phase 3

In this phase, the team determines if the situation warrants a single model or a series of techniques as part of a larger analytic workflow.

phase 3

In this phase, the objective of the data exploration is to understand the relationships among the variables to inform the selection of the variables and methods to understand the problem domain.

phase 3

In this phase, the aim is to capture the most essential predictors and variables rather than considering every possible variable that people think may influence the outcome.

phase 3

In this phase, the team should plan to test a range of variables to include in the model and then focus on the most important and influential variables.

phase 3

In this phase, the team’s main goal is to choose an analytical technique, or a short list of candidate techniques, based on the end goal of the project.

A _______ simply refers to an abstraction from reality.

phase 3

In this phase, the team needs to determine the techniques that are best suited for structured data, unstructured data, or a hybrid approach.

phase 3

In this phase, the team should take care to identify and document the modeling assumptions it is making as it chooses and constructs preliminary models.

phase 4

In this phase, the team needs to develop datasets for training, testing, and production purposes.

training, test

A simple way to think of datasets is to view the ________ dataset for conducting the initial experiments and the _________ datasets for validating an approach once the initial experiments and models have been run,

phase 4

In this phase, an analytical model is developed and fit on the training data and evaluated against the test data.

short

Although modeling logic and techniques required are highly-complex, the actual duration of phase 4 is relatively ____ compared to the time spent preparing data and defining the approaches.

Phases 3 and 4

These two phases tend to move more quickly, although they are more complex from a conceptual standpoint.

phase 4

In this phase, the team assesses the validity of the model and its results, as well as determine if the model accounts for most of the data and has robust predictive power.

phase 4

In this phase, the team refines the models to optimize the results, such as modifying variable inputs or reducing correlated variables where appropriate.

phase 4

In this phase, it is vital to record the results and logic of the model as many decisions that are often made about the data and the approach for the modeling can be easily forgotten once the project is completed.

phase 5

In this phase, after the team executes the model, the team needs to compare the outcomes of the modeling to the criteria established for success and failure.

phase 5

In this phase, the team considers how best to articulate the findings and outcomes to the various team members and stakeholders

phase 5

In this phase, the team needs to determine if it succeeded or failed in its objectives.

phase 5

In this phase, assess the results and identify which data points may have been surprising and which were in line with the hypotheses that were developed in phase 1.

phase 5

In this phase, the team needs to reflect on the implications of the findings and measure the business value.

phase 5

In this phase, the team makes recommendations for future work or improvements to existing processes, and consider what each of the team members and stakeholders need to fulfill their responsibilities.

phase 5

This is the phase to underscore the business benefits of the work and begin making the case to implement the logic into a live production environment.

communicate the results

What is phase 5 otherwise known as?

model building

What is phase 4 otherwise known as?

operationalize

What is phase 6 otherwise known as?

phase 6

In this phase, the team communicates the benefits of the project more broadly and sets up a pilot project to deploy the work in a controlled way before broadening the work to a full enterprise.

phase 6

In this phase, rather than deploying models immediately on a wide-scale basis, the risk can be managed more effectively by undertaking a small scope, pilot deployment before a wide-scale rollout.

phase 6

In this phase, the team is able to learn about the performance and related constraints of the model in a production environment on a small scale and make adjustments before a full deployment.

phase 6

This phase can bring in a new set of team members, usually engineers responsible for the production environment.

phase 6

A part of this phase includes creating a mechanism for performing ongoing monitoring of model accuracy and if accuracy degrades, finding ways to retrain the model.

Data collection

This is the process of collecting, measuring and analyzing different types of information using a set of standard validated techniques.

Primary and secondary data collection

These are the two main methods of data collection in research based on the information that is required.