System Design Roadmap 08 - Reliability and Security Patterns

Reliability pattern goal

Keep the system correct and useful despite partial failure, overload, dependency outages, bugs, and deploy mistakes.

Deployment stamps

Deploy identical isolated copies of an app stack for tenants/regions/scale units. Limits blast radius and simplifies repeated rollout.

Geodes

Deploy active workloads across geographic regions so users route to nearby healthy regions and regional failures do not take down the whole service.

Health endpoint monitoring pattern

Expose endpoints that report service and dependency readiness. Load balancers and orchestrators use them to remove unhealthy instances.

Throttling

Limit request rate or concurrency to protect services. Can be per user, tenant, IP, API key, endpoint, or global.

Bulkhead

Partition resources so one failing area cannot consume everything. Examples: separate thread pools, connection pools, queues, tenants, or regions.

Circuit breaker

Stop calling a failing dependency temporarily after error thresholds. Prevents cascading failure and gives dependencies time to recover.

Compensating transaction

Undo or counteract earlier steps when a distributed workflow cannot use a single transaction. Example: refund after failed booking fulfillment.

Retry pattern

Repeat failed transient operations with limits, exponential backoff, and jitter. Use only for retryable errors and idempotent operations.

Scheduler Agent Supervisor

Coordinate scheduled/distributed work with an agent and supervisor that track progress, retries, and failure handling.

Leader election

Choose one active coordinator among replicas for work that must happen once. Needs leases/timeouts to avoid split-brain leaders.

High availability vs resiliency

High availability focuses on staying up. Resiliency focuses on absorbing and recovering from failures gracefully. They overlap but are not identical.

Federated identity

Trust an external identity provider so users authenticate once and apps rely on tokens/claims. Requires token validation and least privilege.

Gatekeeper pattern

A protected component validates, sanitizes, and controls access before requests reach sensitive services. Useful at boundaries.

Security design basics

Authenticate users/services, authorize least privilege, encrypt in transit/at rest, validate input, audit critical actions, and isolate blast radius.

Secrets management

Store secrets outside code, rotate them, restrict access, audit use, and avoid exposing them in logs, builds, client bundles, or configs.

Rate limiting for abuse control

Limits brute force, scraping, spam, and accidental overload. Combine limits with authentication, reputation, quotas, and monitoring.