System Design

Functional requirements and non functional tell what

Functoinal Requiremsnts tell you which boxes exist in your diagram; nonfunctional Decide for the boxes how many, where they are placed, and how they are connected

The real skill for functional requirements is not listing features it’s…

Cutting them

The non-functional requirements worth asking:

• Scale. How many users, how much traffic? Daily active users, requests per second at peak, and how much data is stored per year. This drives almost everything else.

• Latency. How fast, stated as a percentile. Always ask for p99 rather than an average, because averages hide the slow requests users actually notice.

• Read to write ratio. A hundred reads per write points you at caches and replicas. The reverse points you at queues and write-optimised storage.

• Availability. What downtime is acceptable?

• Consistency. Must every reader see a write immediately, or is a short delay acceptable? This is rarely one answer for the whole system. Account balances need strong consistency; a view counter does not.

• Durability. Is losing data acceptable, and if so, how much? Photos and payments are different from analytics events.

• Security and privacy. Authentication, authorisation, encryption, and any regulation that applies to the data.

• Cost. Rarely stated, always real. It is a good sign when a candidate mentions that a design would be expensive and offers a cheaper option.

When estimating scale you need which things:

1. requests per second at peak, 

2. storage over some horizon, 

3. bandwidth in and out, 

4. The read to write ratio. 

Out of the four things you need to estimate scale, which is the most important and why?

Read to write ratio is the most useful of the four and the one candidates most often skip, because it decides almost every choice that follows. A hundred reads per write points you at caches and replicas. The reverse points you at queues and write-optimised storage.

What are the seven steps to follow for system design interviews?

1. Clarify the requirements 

2. Estimate the scale

3. Define the API 

4. Define the data model 

5. Draw the high-level design 

6. Go deep on two or three things

7. Bottlenecks and failure points

Availability: what the numbers mean per year:

99% = downtime about 3.65 days per year

99.9% = about 8.8 hours

99.99% = about 53 minutes

99.999% = about 5 minutes

What does more availability cost you?

Every extra nine costs real money and real complexity. A candidate who says "99.999%" without being asked has usually not thought about the price.

A requirement of very high read volume pushes you toward….

caching, read replicas, CDN

A requirement of low latency for a global audience pushes you toward….

CDN, multiple regions, data near users

A requirement of high availability pushes you toward….

Redundancy everywhere, no single points of failure, multi-zone

A requirement of strong consistency pushes you toward….

A single primary, synchronous replication, accepting higher latency

A requirement of high write volume pushes you toward….

Queues, batching, sharding, LSM-based storage

A requirement of large data volume pushes you toward….

Partitioning, object storage, tiering cold data

A requirement of spikey traffic pushes you toward….

Queues to absorb bursts, autoscaling

Time to memorize (a day, a million a day, a billion a day)

• A day → (86,400) round to 100,000 seconds

• 1 million a day → ~12 per second

• 1 billion a day → 12,000 per second

Sizes to memorize (character, short post, photo, a minute of 1080p)

• A character → 1 byte

• A short post → ~300 bytes

• A photo → ~200 KB to 2 MB

• A minute of 1080p → ~30MB

Powers of two to memorize (2^10, 2^20, 2^30, 2^40)

• 2^10 → thousand, KB

• 2^20 → million, MB

• 2^30 → billion, GB

• 2^40 → trillion, TB

Sequence of steps to start estimating

Daily active users → actions per user (post, views, upload) → requests per second (then multiply for peak)

Storage (from totals: every item ever kept based on actions per user)

bandwidth (from concurrency; who is active now)

Note: Notice the split at the bottom, because it is where most mistakes happen. Storage comes from totals. Bandwidth comes from concurrency.

Example of estimating load

A social platform has 100 million daily active users posting 10 times a day.

100M users x 10 posts        = 1B posts per day

1B / 100,000 seconds         = ~10,000 writes per second, average

Average is not what you design for. Traffic peaks, and a factor of two to three is a reasonable default to state:

10,000 x 3                   = ~30,000 writes per second at peak

Then ask for the read-to-write ratio, because it is the most useful number in the whole exercise. At a hundred reads per write you get three million reads per second, and that single figure is what justifies caching, replicas and a CDN for the rest of the interview.

Example of estimating storage

A photo app has 500 million users uploading 2 photos a day at 2 MB each.

500M x 2 x 2 MB              = 2 PB per day

2 PB x 365                   = ~730 PB per year

Then adjust for what you actually keep. Three replicas triples it. Thumbnails add ten to twenty percent. Compression may reduce it somewhat. The first number is the one that matters, because 2 PB a day already tells you this is object storage and never a database.

example of estimating bandwidth

A video service has 10 million daily users watching about an hour each, at 4 Mbps for 1080p.

The common mistake is multiplying 10 million by 4 Mbps. That gives 40 Tbps and assumes everybody watches at the same instant, which never happens. Find the concurrency first:

10M users x 1 hour           = 10M viewing-hours per day

10M / 24 hours               = ~420K watching at any moment

420K x 3 (peak)              = ~1.25M concurrent viewers

1.25M x 4 Mbps               = ~5 Tbps at peak

Still enormous, and that is the point. The number tells you immediately that serving this from your own origin is not an option and that a CDN is doing nearly all the work.

Estimations: Three ratios are worth remembering:

• Memory is about a hundred thousand times faster than a disk seek. This is why caching exists and why it is usually the single change with the largest effect.

• An SSD read is roughly a hundred times faster than a spinning disk seek. This is why storage engine choice matters at high read volume.

• Crossing an ocean costs somewhere near 100 ms, and no engineering removes it. This is why multi-region designs place data near users instead of trying to remove the delay with complex engineering.

Load balancer or API gateway?

Both, usually. They do different jobs: the load balancer picks a machine, the gateway picks a service and applies policy. A typical arrangement is a load balancer in front of several gateway instances, and more load balancing behind the gateway.

Your one server is at 90 percent CPU, and traffic is still growing. You have exactly two options.

Buy a bigger machine, or add a second machine.

pros of vertical scaling

• No code changes. This is the real advantage, and it is easy to undervalue. No load balancer to add, no session store to introduce, no data to split.

• No distributed systems problems. One machine holds the only copy of the data, so copies cannot disagree. Transactions, joins, and locks keep working exactly as they did.

• Simple operations. One machine to deploy, monitor, and debug. One set of logs.

• Cheap in engineering time. Engineer hours are a real cost, even though they never appear on the infrastructure bill.

cons of vertical scaling

• A hard ceiling. There is a largest machine for sale. Once you run on it, more money buys nothing.

• Price grows faster than capacity. Ordinary hardware is cheap, and top-end hardware is not. Doubling the cores usually more than doubles the price.

• No redundancy. Redundancy means having a second copy that can take over after a failure. A bigger server is still one server. When it fails, everything is down.

• Resizing usually means downtime, or at least a restart and a switch to a standby machine.

pros of horizontal scaling

• No practical ceiling. Need more capacity, add more machines. Every large system is built this way.

• Redundancy comes built in. Lose one server out of ten, and you lose ten percent of capacity, not the service.

• Cheaper per unit at scale. Many ordinary machines cost less per unit of capacity than one giant machine.

• You can scale down too. Ten machines at peak and three overnight. Capacity that comes in units works in both directions.

• Safer deploys. Update one machine at a time while the others keep serving.

cons of horizontal scaling

• A load balancer to run, and it must be made highly available itself.

• State becomes a problem. State is the data a server remembers between requests, such as a user session. If a server keeps state in its own memory, every later request must return to that exact machine. So your servers must become stateless, keeping no user data between requests. Stateful vs Stateless Architecture covers this constraint in depth.

• Data must be copied or split. The application tier multiplies easily. The data tier does not, and the next section explains why.

• Machine failures become normal events. With fifty machines, something is always broken, and the network between your own machines can fail too. The system must keep working through both.

• Harder operations. A request may touch any of fifty machines. Deploys, monitoring, logs, and debugging all get harder.

• Some work does not split. A single long computation that cannot be divided runs no faster on a hundred machines than on one. If every write must pass through one machine, that machine sets the limit, no matter how many others you add.

Stateless application servers scale ______ with almost no extra work.

Stateless application servers scale horizontally with almost no extra work. Put a load balancer in front and run as many as you want. If a server holds nothing the other servers need, there is no hard problem left.

Real systems rarely pick just vertical or horizontal scaling and stop. They follow a path, and the order matters.

1. One machine. Everything runs on it.

2. Scale up. A bigger machine, at the cost of money and no engineering time.

3. Scale the app tier out. Stateless servers behind a load balancer, sessions in a shared store, read replicas for the database.

4. Shard the database. Only when write volume outgrows the biggest sensible primary.

The rule inside the path for how to scale up:

The rule inside the path: scale up first, because it costs no engineering time. And design so you can scale out later, because eventually you must. Concretely, keep application servers stateless from the first day, even while you run only one. Then adding the second server is a configuration change, not a rewrite.

Scale vertical or horizontal for Early, small, or urgent?

Early, small, or urgent: scale up. It buys months of growth for zero engineering time.

Scale vertical or horizontal for A stateless service with growing traffic?

A stateless service with growing traffic: scale out. This is the easy case, so take it.

Scale vertical or horizontal for The data tier?

The data tier: scale up first, add read replicas for reads, and shard only when writes demand it.

Scale vertical or horizontal for Availability is a requirement?

Availability is a requirement: one machine cannot survive its own failure. You need at least two of everything that must stay up, which means scaling out

Scale vertical or horizontal for Work that cannot be divided?

• Work that cannot be divided: stays vertical. More machines do not help a job that only one machine can run.

Estimate the scale for pastebin: new pastes per day, and the ratio of reads to writes. Assume one million new pastes a day and a 5:1 ratio between reads and writes, so five million reads a day. What is Load, Storage and Bandwidth?

Load.

1M / (24 hours x 3600 seconds)   = ~12 new pastes per second

5M / (24 hours x 3600 seconds)   = ~58 reads per second

Storage. The limit is 10 MB, but most pastes are small. Source code, configs, and logs are usually a few kilobytes. Assume 10 KB per paste on average.

1M x 10KB                        = 10 GB per day

10 GB x 365 days x 10 years      = ~36 TB

We also store a seven-character key for each paste. Ten years of pastes is 3.6 billion keys.

1M x 365 days x 10 years         = 3.6 billion pastes

3.6B x 7 bytes                   = ~25 GB

25 GB is negligible next to 36 TB. To keep a safety margin, assume a 70 percent capacity model: we never want to use more than 70 percent of the storage we own. That raises the ten-year target to about 51.4 TB.

36 TB / 0.7                      = ~51.4 TB

Bandwidth.

12 writes x 10KB                 = ~120 KB per second in

58 reads x 10KB                  = ~0.6 MB per second out

Pastebin: If 20 percent of pastes produce 80 percent of the traffic, we want those pastes in _____.

In memory. (cache)