Chapter 3: Scaling - Distributed Systems And Design NodeJS

Splitting services

Running redundant copies of a service is essential for high availability and handling workload limits.

1. High Availability: If a single instance crashes, consumers may be unable to function until it restarts. Running multiple instances ensures that a failure won’t completely disrupt operations.

2. Performance & Scalability: A single Node.js instance has a limit on the number of requests it can handle. As workload increases (e.g., serialization, computation), throughput drops significantly. Distributing the load across multiple instances prevents a single process from being overwhelmed.

Tools for splitting services

• There are a few tools available for splitting up work. “The Cluster Module” looks at a built-in module that makes it easy to run redundant copies of application code on the same server.

• “Reverse Proxies with HAProxy” runs multiple redundant copies of a service using an external tool—allowing them to run on different machines.

• “SLA and Load Testing” looks at how to understand the load that a service can handle by examining benchmarks, which can be used to determine the number of instances it should scale to.

The Cluster Module

Node.js provides the cluster module to allow running multiple copies of a Node.js application on the same machine, dispatching incoming network messages to the copies.

This module is similar to the child_process module, which provides a fork() method for spawning Node.js sub processes; the main difference is the added mechanism for routing incoming requests. (check img)

Understanding Cluster Module

In Node.js clustering, a master process creates multiple worker processes to handle incoming requests.

Instead of each worker listening on a port, they send a message to the master, which manages the port and distributes requests among them.

If workers use port 0 (random selection), the master assigns a shared random port, ensuring efficient request handling. This setup improves scalability and load balancing in backend systems.

Request Dispatching

On macOS and Linux, requests are distributed evenly among workers. On Windows, requests go to the worker that seems the least busy.

(for later understanding: 80-82)

Cluster Shortcomings

• it’s better to keep the master process as simple as possible. Example 3-1 proves that there’s no reason for a master to load an HTTP framework or consume another database connection. Logic could be built into the master to restart failed workers, but the master itself can’t be restarted as easily.

• The cluster module operates at Layer 4 (TCP/UDP) and isn't aware of Layer 7 protocols like HTTP. This means:

  • For HTTP requests, each new request gets dispatched to a different worker once the TCP connection closes.

  • With gRPC over HTTP/2, connections stay open longer, preventing requests from being evenly distributed.

  • This can lead to one worker handling most of the load, defeating the purpose of clustering.

(requires more understanding and experimenting 82-86)

Why using a Reverse Proxy (With Nginix)

Using Nginx as a reverse proxy offers several advantages in various aspects of web server performance and security:

• SSL Encryption: Offloads SSL/TLS encryption from backend servers, improving efficiency while securing client-server communication.

• Buffering: Sends data in chunks, making it ideal for streaming files and videos. This reduces latency and ensures that clients receive content smoothly without overloading backend servers.

• Recovery: Provides error handling mechanisms, ensuring smooth failover and customized error pages for a better user experience.

• Load Balancing: Distributes incoming traffic across multiple backend servers using different algorithms (round-robin, least connections, etc.), enhancing scalability and reliability.

• Enterprise Routing: Directs specific requests to designated servers based on predefined rules, Ex: Direct routes that coming from fetching cosmetic products to a server that only deals with cosmetic products. This improves efficiency for high-traffic services like search functions.

• Gzip Compression: Compresses text-based responses (HTML, CSS, JavaScript) before transmission, reducing bandwidth usage and speeding up page load times.

• Caching Static Content: Stores frequently accessed assets like images, stylesheets, and scripts to minimize unnecessary server load and reduce response times.

Reverse Proxies with HAProxy

• A reverse proxy is a tool that accepts a request from a client, forwards it to a server, takes the response from the server, and sends it back to the client.

• Reverse proxies often operate at either Layer 4, such as TCP, or Layer 7, via HTTP.

• One of the features it provides is that of load balancing. A reverse proxy can accept an incoming request and forward it to one of several servers before replying with the response to the client.

• most performance-critical applications. Because of this you should almost always use some form of reverse proxy between your Node.js applications and the internet.

(86-88)

HAProxy (Popular alternative is Nginx.)

HAProxy is a very performant open source reverse proxy that works with both Layer 4 and Layer 7 protocols. It’s written in C and is designed to be stable and use minimal resources, offloading as much processing as possible to the kernel. Like JavaScript, HAProxy is event driven and single threaded.

HTTP benchmarking

Autocannon is a high-performance HTTP benchmarking tool for Node.js, designed to test the load capacity of web applications. It helps developers measure how well their APIs or web servers handle high traffic by simulating multiple concurrent requests.

Key Uses of Autocannon:

• Load Testing: Simulates thousands of requests per second to evaluate server performance.

• Benchmarking: Compares different configurations to optimize response times.

• Stress Testing: Identifies bottlenecks by pushing a server to its limits.

• Performance Monitoring: Provides detailed metrics on latency, requests per second, and throughput.

• Pipelining Support: Tests how well a server handles multiple requests in a single connection.

(For better Autocannon usage 104-)

SLOs (Service Level Agreement)

A Service-Level Objective (SLO) is a key metric used in site reliability engineering (SRE) and service agreements to define the expected performance of a service. It represents a target value or range for a service level indicator (SLI), such as uptime, response time, or availability.

SLOs are often part of Service-Level Agreements (SLAs), which are formal contracts between a service provider and a customer. While SLAs define the overall agreement, SLOs focus on specific measurable aspects of service performance.

For example, an SLO might state that a web service should maintain 99.9% uptime over a given period. If the service fails to meet this target, it could trigger corrective actions or impact customer satisfaction.