Software Development and Data Schemas

Why software development is considered complex

Because teams can be large, members often collaborate across countries/continents, the software being built is sophisticated and custom, and a team's reputation depends on building robust, error-free systems.

Typical codebase size

It's not unusual for a single system to contain 50,000+ lines of code, not counting external libraries.

Why use tooling in software development

Tooling provides standardisation (so by convention everyone knows where things are) and productivity (tools handle common tasks so developers can focus on the actual software).

Common categories of development tooling

Package managers, virtual environments, version control, formatters, test frameworks, and task runners.

Package manager (definition)

A tool that adds, removes, restores, and upgrades software dependencies, including transitive dependencies.

What a package manager automates when adding a dependency

Resolving conflicts between dependencies, choosing the right dependency version for your machine/OS, and tracking which dependencies were requested.

What a package manager automates when removing a dependency

Undoing changes introduced by the dependency, removing its transitive dependencies, and running any scripts needed to reconfigure the setup.

PIP

A Python package manager used to install Python packages.

Conda

A package and environment manager that works for Python and other languages.

Namespace

A container used at runtime to hold Python symbols (such as variables, functions, classes, or modules) and their values.

Ways to add an entry to a namespace

Assigning a variable, defining a function or class, or importing a module.

Python module

A directory containing Python scripts, most often representing a reusable code library.

init.py

The script that initializes a module's namespace when it's loaded — it can set variables, define functions, import entities from other scripts, and check for required dependencies.

How Python modules can be nested

A module can have subdirectories that define additional modules within its own namespace (e.g. xml.dom, xml.parsers, xml.sax, and xml.etree are all modules within the "xml" module).

What an installable Python package contains

The module's directory (with all its scripts/subdirectories) plus a setup.py script that drives installation.

setup.py

A script that specifies a module's name, version, and required dependencies (optionally with min/max versions), and is used to build and install/update the package.

How Python locates a module

Similar to how Unix finds an executable: a series of directories is searched in order for a matching, accessible file, and the first match found is used.

PYTHONPATH

An environment variable listing directories that Python searches, in order, to locate a module by name before falling back to the default install locations.

Downside of a large PYTHONPATH

As more directories are added, they all must be scanned on every import, which can slow down module loading.

Core problem with installing Python modules individually

Managing all of their dependencies (and the dependencies of those dependencies) by hand.

Solution 1: common directory (for managing modules)

Store all modules in one shared directory so only a single path needs to be added to PYTHONPATH; the caveat is every module and its dependencies must still be downloaded, built, and installed by hand.

Solution 2: PIP with a common directory

Using pip to install into a chosen directory (via --prefix) automatically resolves and installs Python-level dependencies recursively; --ignore-installed forces it to ignore default modules already present.

Problem with Solution 2 (PIP + common directory)

PIP only understands Python-level dependencies, so modules with compiled components can be installed successfully but crash if they were built against system libraries (e.g. a different glibc version) not present on the machine.

How Python virtual environments came about

Someone realized that recreating the specific directory structure Python uses to search for its script library lets any directory be treated as a standalone Python installation.

Solution 3: Virtual Environments (virtualenv)

A tool that turns a directory into a self-contained Python installation: no PYTHONPATH is needed, pip installs into the container, and container modules override the base installation for anything not present in it.

Limitation still present in Solution 3 (virtualenv)

It inherits the same problem as plain pip: modules with compiled components built against libraries not present on the system will still crash.

Solution 4: (Ana)conda

A separate package management system where each conda container is its own virtual environment; pip can still manage pure Python modules, while conda specifically manages modules with compiled components (including things like CUDA libraries).

requirements.txt

A text file listing all Python packages (and their versions) needed for a project, allowing others or deployment servers to reproduce the same environment.

pip freeze

The command used to generate a requirements.txt file listing currently installed packages and versions (pip freeze > requirements.txt).

Principle of isolation (Python virtual environments)

Virtual environments let packages be installed in an isolated location for a particular application rather than globally; each has its own installation directory and doesn't share libraries with other environments.

Version Control (definition)

A way of storing the state of files over time and being able to return them to how they were at earlier points in time.

Reasons to use Version Control

Being able to undo mistakes, tracking history (what changed, when, and by whom), reusing or reverting to good ideas, disaster recovery, and enabling collaboration between people.

Git

An open source, distributed version control system.

Git commits

"Snapshots" of the entire project taken at specific moments, which together form a linear history.

Why Git is called a distributed system

Because every developer has a full copy of the entire repository history on their own local machine.

Task Runner — what it can do

Depending on the ecosystem, it can create a new project from a template, build a project, publish it in final form, manage dependencies, run the project, and check it using a test framework.

Formatters (definition)

Tools that automatically format source code according to predefined style guidelines, applying consistent indentation, spacing, line breaks, and other rules.

Benefit of automated code formatting for teams

It ensures all code follows the same standard, reducing cognitive load and letting code reviews focus on logic rather than style.

Idempotent formatter

A formatter that causes no further changes if it's run again on code that's already been formatted.

Black

The "uncompromising" Python code formatter — considered opinionated because it offers few configuration options and enforces one specific, PEP 8-compliant style.

PEP 8

The official style guide for Python code; since code is read far more often than it's written, it focuses on consistency to improve readability.

Test Framework (definition)

Consists of software libraries/packages that make it easy to write tests (unit tests, integration tests, mocking) plus a test runner that executes tests and reports results.

Unit tests

Tests that check discrete, isolated pieces of functionality.

Integration tests

Whole-system tests that check overall system functionality.

Mocking

A testing technique (part of a test framework) used to simulate parts of a system so functionality can be tested in isolation.

Arrange-Act-Assert pattern

The standard pattern for writing unit tests: arrange the environment, act by calling the function/method under test, then assert the result matches what's expected.

Arrange (in Arrange-Act-Assert)

The step where you set up the inputs and expected result / put a particular environment in place.

Act (in Arrange-Act-Assert)

The step where you call the function or method being tested.

Assert (in Arrange-Act-Assert)

The step where you check that the actual result matches the expected result; if the assertion fails, the test fails, otherwise it passes.

FIRST principles of unit tests

Unit tests should be Fast, Independent/Isolated, Repeatable, Self-checking, and Thorough.

Fast (unit test principle)

Each test should execute quickly, typically taking only a few milliseconds to run.

Independent / Isolated (unit test principle)

A test must not depend on the result of any other test, nor on any other test having run first.

Repeatable (unit test principle)

A test must reliably give the same result no matter when or where it's executed.

Self-checking (unit test principle)

A test must determine pass/fail itself using clear assertions, without any human judgment or manual inspection.

Thorough (unit test principle)

Tests should cover all likely cases and as many edge cases as possible.

Schema (in MongoDB data modelling)

Defined at the application level rather than enforced by the database; it comes from the needs of the application and should evolve as the application changes.

Schema Design Considerations

The queries and specific data your application needs, how your application reads and writes data (read/write patterns), and the relationships between your data (linked or embedded).

Embedded relationship (schema design)

Related data (e.g. comments) is stored as a nested sub-document/array inside the parent document (e.g. inside the blog post itself).

Linked relationship (schema design)

Related data is stored in a separate collection and connected via a reference field (e.g. comments store a blog_id pointing back to the blog post).

One-to-One (1-1) relationship

A relationship where a single document of one type relates to exactly one document of another type.

One-to-Many (1-N) relationship

A relationship where a single document relates to multiple documents of another type (e.g. one customer to many orders).

Many-to-Many (N-N) relationship

A relationship where multiple documents of one type relate to multiple documents of another type (e.g. many invoices reference many products).

Reasons to embed data

Good for one-to-one and one-to-many relationships, for data that should be deleted together by default, and for integrity on read operations (retrieving related data atomically in one operation).

Reasons to link (reference) data

Good when the "many" side of a relationship is a huge/unbounded number, for integrity on write operations in many-to-many relationships, and when one piece of data is used far more often than the related piece and memory is a concern.

Modelling Methodology (four stages)

Workload, Relationships, Patterns, and Schema.

Workload stage (modelling methodology)

Size the data, and quantify and qualify the operations the application performs on it.

Relationships stage (modelling methodology)

Identify and quantify relationships between entities, then decide whether to embed or link them.

Patterns stage (modelling methodology)

Recognize which schema design patterns apply, then apply them.

Schema stage (modelling methodology)

The output of the methodology: the resulting collections, fields, document shapes, and applied patterns.

Flexible Methodology goals

Depending on the project's priority — Simplicity, Simplicity and Performance, or Performance — the depth of workload analysis, degree of embedding vs linking, and number of patterns applied all increase accordingly.

Why use Schema Design Patterns

They provide reusable techniques for transforming a schema, give teams a common language for discussing design, and can be applied within a broader modelling methodology.

Data Driven Dynamic Schema

A schema where field names themselves are data values rather than fixed labels; it's an unfamiliar concept for many designers, requires a truly dynamic coding approach, but can be clean and performant.

Fixing a data-driven schema anti-pattern

Instead of using data values as field names (e.g. {john: {score: 25}}), restructure as an array of objects with explicit fields (e.g. [{player: "john", score: 25}]) so the data is easier to query.

Attribute Pattern

Uses dynamic keys to represent variable attributes instead of adding many separate columns/fields.

Subset Pattern

Stores only a subset of data (the most frequently accessed portion, e.g. the last 5 comments) in the main document for fast access, rather than the entire related array.

Outlier Pattern

Stores normal-sized documents together as usual, but moves unusually large documents elsewhere so they don't affect the rest of the collection.

Bucket Pattern

Groups related events together into one document, array, or subdocument (commonly used for time-series/IoT data).

Computed Pattern

Precomputes and stores values that would be expensive to calculate at query time, trading storage for faster reads.

Other named schema design patterns (not detailed in the overview)

Approximation, Document Versioning, Extended Reference, Preallocated, Polymorphic, Schema Versioning, and Tree.