Untitled

What is Machine Learning?

• ML is about extracting knowledge from data. It sits at the intersection of statistics, artificial intelligence, and computer science.

• Also known as predictive analytics or statistical learning.

• Ubiquity: used in everyday life for recommendations (movies, foods, products), personalized online radio, recognizing friends in photos, and at the core of many websites and devices (e.g., Facebook, Amazon, Netflix).

• Impact beyond apps: data-driven research in science (stars, distant planets, new particles, DNA analysis, personalized cancer treatments).

• Goal of this chapter: explain why ML is popular, what problems ML can solve, and how to build your first ML model with key concepts along the way.

Why Machine Learning?

• Early systems used hand-coded rules (if-else) to process data (example: spam filter using a blacklist of words).

• Disadvantages of hand-coded rules:

  • The logic is domain- and task-specific; changing the task may require rewriting the system.

  • Designing rules requires deep human expertise about how decisions should be made.

• Face detection example: pixels are perceived differently by computers than by humans, so crafting a good rule set is hard.

  • With ML, you can train a program by showing it many images of faces, allowing the algorithm to learn the necessary features.

Problems ML Can Solve

• Supervised learning: learns from input-output pairs (training data) to predict outputs for new inputs.

• Concept: a “teacher” provides supervision via the desired outputs for each training example.

• Spam classification example: given many emails (inputs) and labels (spam vs not spam), the model predicts whether a new email is spam.

• If the application can be formulated as supervised learning and a labeled dataset is available, ML is likely to help.

• Unsupervised learning (only inputs, no outputs): harder to understand and evaluate but useful for discovering structure in data.

• Examples of unsupervised tasks:

  • Identifying topics in blog posts (no known outputs; topics unknown a priori).

  • Segmenting customers into groups with similar preferences (no predefined groups or counts).

  • Detecting abnormal access patterns to a website (no prior abnormal examples).

Data Representation: Samples and Features

• Data is often represented as a table:

  • Each row is a sample (data point) to reason about (e.g., an email, a customer, a transaction).

  • Each column is a feature describing that sample (e.g., age, location, transaction amount; or grayscale pixel values for an image).

• Samples are called data points; features are the properties describing them.

• Terms in scikit-learn conventions:

  • X = data matrix (samples × features), typically denoted as a two-dimensional array.

  • y = target vector (labels) for each sample, typically one-dimensional.

• Important caution: a model cannot predict something that is not represented in the data. Example: predicting gender from only a last name is impossible if the data lacks any gender-related feature.

• Feature engineering/representation is crucial; good features enable better predictions.

Knowing Your Task and Knowing Your Data

• Before building a model, understand the data and how it relates to the task.

• Key questions to consider:

  • What question(s) am I trying to answer? Do I think the data can answer it?

  • How should I phrase the question as a ML problem?

  • Have I collected enough data to represent the problem?

  • What features were extracted, and will they enable the right predictions?

  • How will I measure success in the application?

  • How will the ML solution interact with other parts of the product or research?

• Big picture: algorithms are only one part of a larger process; avoid solving the wrong problem by focusing on assumptions and goals.

Why Python? The ML Ecosystem

• Python is popular for data science due to its balance of general-purpose programming and domain-specific tooling.

• Benefits:

  • Interactive exploration via terminal/Jupyter Notebook.

  • Extensive libraries for data loading, visualization, statistics, NLP, image processing, etc.

  • Supports rapid iteration and integration into larger systems (GUIs, web services).

• scikit-learn (sklearn): a leading open-source ML library in Python with many state-of-the-art algorithms and thorough documentation.

• scikit-learn is built on top of NumPy and SciPy and works well with other scientific Python tools.

Essential Libraries and Tools

• Core stack:

  • NumPy: foundational for scientific computing; ndarray is the core data structure used by scikit-learn. All data to scikit-learn is represented as NumPy arrays.

  • SciPy: collection of scientific computing tools (advanced linear algebra, optimization, signal processing, statistical distributions); includes scipy.sparse for sparse matrices.

  • matplotlib: plotting library for publication-quality visualizations.

  • pandas: data wrangling with DataFrame (like a table, with potentially different dtypes per column).

  • Jupyter Notebook: browser-based interactive environment for combining code, text, and visuals; ideal for exploratory analysis.

• Common integration: data flows through NumPy arrays, with SciPy/pandas providing processing and visualization capabilities.

• mglearn: accompanying library with utility functions and helpers used in the book to simplify plotting and data loading.

• Quick-start via pip (if you already have Python):

  • $ pip install numpy scipy matplotlib ipython scikit-learn pandas

• Data and code examples in this book are designed to work with standard Python scientific stack (NumPy, SciPy, Matplotlib, Pandas, scikit-learn).

Jupyter Notebook, NumPy, SciPy, Matplotlib, and Pandas in this Book

• Jupyter Notebook: interactive browser-based environment; allows combining code, text, and images; used to present code and results in this book.

• NumPy: fundamental arrays and operations; ndarray is the core data structure; data used by scikit-learn is converted to NumPy arrays.

• SciPy: adds advanced math routines; scipy.sparse provides sparse matrices (useful when data is mostly zeros).

  • Example representations:

  • Dense to sparse CSR: CSR format stores only nonzero entries with their indices.

  • COO format can also be used to construct sparse matrices.

• matplotlib: plotting library; example shows simple sine plot and other figures.

• pandas: DataFrame for tabular data; supports SQL-like queries and joins; can ingest from various formats (CSV, Excel, SQL, etc.).

• mglearn: utility library used in examples (not essential to the core material).

Python Versions and Versions Used in This Book

• Two major Python versions in use: Python 2.x (2.7) and Python 3.x (latest at the time: 3.5).

• Python 2 is no longer actively developed; Python 3 is recommended for new work.

• The book uses versions compatible with scikit-learn >= 0.18; the model_selection module was added in 0.18.

• Example environment snapshot from the book:

  • Python version: $ext{Python version: } 3.5.2$

  • pandas version: $ext{pandas version: } 0.18.1$

  • matplotlib version: $ext{matplotlib version: } 1.5.1$

  • NumPy version: $ext{NumPy version: } 1.11.1$

  • SciPy version: $ext{SciPy version: } 0.17.1$

  • IPython version: $ext{IPython version: } 5.1.0$

  • scikit-learn version: $ext{scikit-learn version: } 0.18$

• The exact versions are not critical, but use a recent scikit-learn (>= 0.18).

• The book provides code that uses these imports and the following pattern: traintestsplit, KNeighborsClassifier, etc., and emphasizes that the fit/predict/score methods are the core interface for supervised models.

A First Application: Classifying Iris Species

• Data setup:

  • Iris dataset used for a simple supervised learning task: predict iris species from four measurements.

  • Measurements (features):

  • sepal length (cm)

  • sepal width (cm)

  • petal length (cm)

  • petal width (cm)

  • Targets (labels): species names, encoded as integers 0, 1, 2 corresponding to

  • 0: setosa

  • 1: versicolor

  • 2: virginica

• Loading the dataset:

  • irisdataset = loadiris()

  • iris_dataset is a Bunch object (similar to a dict) with keys:

  • target_names: array(['setosa', 'versicolor', 'virginica'])

  • feature_names: array(['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)'])

  • DESCR: description of the dataset

  • data: NumPy array of shape $(150,4)$ containing measurements

  • target: NumPy array of shape $(150,)$ containing integer labels (0,1,2)

• Data understanding:

  • The data array shape is $(150, 4)$: 150 samples, 4 features each.

  • The target array shape is $(150,)$: one label per sample.

  • Example: the first five rows of data show patterns like [5.1, 3.5, 1.4, 0.2].

  • Target values are integers 0–2, with meanings given by target_names.

• Data description and interpretation:

  • The iris dataset is a classic, often used to illustrate ML workflows.

  • The data representation: X (data) = features, y (target) = labels.

  • The dataset contains 3 classes (setosa, versicolor, virginica), so it is a 3-class classification problem.

• Training/testing split:

  • To evaluate generalization, we split the data into a training set and a test set.

  • Function: traintestsplit from sklearn.model_selection shuffles and splits the data.

  • In the example, 75% is used for training and 25% for testing (train, test split).

  • The split is randomized with a fixed seed for reproducibility using random_state=0.

  • Resulting shapes:

  • Xtrain: $(112, 4)$, ytrain: $(112,)$

  • Xtest: $(38, 4)$, ytest: $(38,)$

• Why the split matters:

  • Training data is used to build the model; test data is used to evaluate generalization to unseen data.

  • If we evaluated on the training data, the model could simply memorize it and give overly optimistic results.

• Visualizing the data (pair plot):

  • A pair plot (scatter_matrix) across feature pairs, colored by class, helps assess separability.

  • Diagonal contains histograms of each feature.

  • Observation: three iris classes appear relatively well separated using sepal and petal measurements, suggesting a ML model could learn to distinguish them.

Building Your First Model: k-Nearest Neighbors (KNN)

• Choice of algorithm: k-Nearest Neighbors (KNN) is chosen for its simplicity and interpretability.

• Idea:

  • For a new data point, find the closest training point(s) and assign the label by majority vote among the nearest neighbors.

  • With k = 1, use the single closest neighbor.

• Implementation details:

  • The KNN classifier is implemented as KNeighborsClassifier in sklearn.neighbors.

  • Instantiate with parameters, e.g., n_neighbors = 1.

  • The model object encapsulates the training data and the algorithm to make predictions.

• Training the model:

  • knn = KNeighborsClassifier(n_neighbors=1)

  • knn.fit(Xtrain, ytrain)

  • The fit method returns the classifier (estimator) itself; the representation shows default parameters and n_neighbors=1 in use.

• Making a prediction:

  • Example new iris: X_new = [[5, 2.9, 1, 0.2]]

  • prediction = knn.predict(X_new) → [0]

  • Predicted class name: irisdataset['targetnames'][prediction] → ['setosa']

• Evaluation on the test set:

  • ypred = knn.predict(Xtest)

  • Test set score (accuracy): $ext{Accuracy} = rac{1}{|D<em>{ ext{test}}|} \sum</em>{i \,=\, 1}^{|D<em>{ ext{test}}|} \, \mathbf{1}(\hat{y}</em>i = y_i)$

  • For this model: score ≈ $0.97\, (97\%)$

  • Alternative calculation: $ext{Accuracy} = \text{mean}(\,y<em>{ ext{pred}} == y</em>{ ext{test}}\,)$

  • Using scikit-learn's convenience method: knn.score(Xtest, ytest) = 0.97

• Interpretation:

  • A test set accuracy of about 0.97 indicates that the model correctly predicts the iris species for about 97% of the test samples.

  • This suggests the model is trustworthy for this task, though caveats about generalization still apply (future data may differ).

• Summary of the model in code form (high-level):

  • Split data: Xtrain, Xtest, ytrain, ytest = traintestsplit(irisdataset['data'], irisdataset['target'], random_state=0)

  • Create model: knn = KNeighborsClassifier(n_neighbors=1)

  • Train: knn.fit(Xtrain, ytrain)

  • Evaluate: score = knn.score(Xtest, ytest)

  • Output: Test set score: {:.2f} -> 0.97

Key Takeaways from the Iris First Application

• The Iris dataset provides a concrete, simple starting point for supervised learning with a small feature set and three classes.

• Data representation in scikit-learn uses:

  • X: 2D array of shape $(n ext{ samples}, n ext{ features})$.

  • y: 1D array of length $n ext{ samples}$ with integer-encoded class labels.

• The process demonstrated:

  • Load data and inspect its structure and metadata (keys, DESCR, targetnames, featurenames).

  • Understand data shapes and feature meanings before modeling.

  • Visualize data to gauge separability (pair plot).

  • Split data into training and testing sets to evaluate generalization.

  • Train a simple, interpretable model (KNN with $k=1$).

  • Predict on new samples and map numeric predictions to human-readable class names.

  • Evaluate with accuracy on a held-out test set and interpret the result.

• Core ML interfaces illustrated:

  • fit(Xtrain, ytrain): train the model on the training data.

  • predict(X_new): output predictions for new samples.

  • score(Xtest, ytest) or computing accuracy via mean of correct predictions.

• This workflow (load data, split, train, predict, evaluate) is the common pattern for supervised ML tasks in scikit-learn.

Quick Reference: Core Terms and Concepts (defined in this chapter)

• Sample: a single data point (row) in the dataset.

• Feature: a column describing an attribute of a sample (e.g., sepal length).

• X: data matrix of shape $(m, n)$, where $m$ is the number of samples and $n$ is the number of features.

• y: target vector of length $m$ containing labels for each sample.

• Class: a possible output category in classification (e.g., 0, 1, 2 for the iris dataset).

• Label: the true class for a sample.

• Estimator: a trained ML model instance in sklearn (e.g., KNeighborsClassifier).

• Fit: training the model on data.

• Predict: generating predictions for new data.

• Score/Accuracy: fraction of correct predictions on a test set.

• Supervised learning: learning from input-output pairs with labels.

• Unsupervised learning: learning from inputs only, without labels.

• Feature engineering: creating or selecting features to improve model performance.

• Data matrix: $X \in \mathbb{R}^{m \times n}$

• Labels: $y \in \{1, \\ldots, C\}^m$ or $\{0,1,2,\ldots, C-1\}^m$ depending on encoding.

• Pairwise prediction concept: $f(x) = y$ where $f$ is the learned mapping from inputs $x$ to outputs $y$.

This chapter has laid the foundation for supervised learning with a concrete, end-to-end example (Iris) and introduced the basic workflow, tools, and concepts that will be used throughout the book.