1/28
Looks like no tags are added yet.
Formats for Encoding Data
Programs usually work with data in (at least) two different representations:
In memory, data is kept in objects, structs, lists, arrays, hash tables, trees, and so on. These data structures are optimized for efficient access and manipulation by the CPU (typically using pointers).
When you want to write data to a file or send it over the network, you have to encode it as some kind of self-contained sequence of bytes (for example, a JSON document). Since a pointer wouldn’t make sense to any other process, this sequence-of-bytes representation looks quite different from the data structures that are normally used in memory.
Thus, we need some kind of translation between the two representations. The translation from the in-memory representation to a byte sequence is called encoding (also known as serialization or marshalling), and the reverse is called decoding (parsing, deserialization, unmarshalling).ii

Language-Specific formats (JSON.stringy, JSON.parse & V8.serialize)
Many programming languages come with built-in support for encoding in-memory objects into byte sequences. For example, Java has java.io.Serializable. Javascript JSON methods & Nodejs V8 methods. Many third-party libraries also exist, such as Kryo for Java [4].
Flaws of Language-Specific formats (JSON.stringy, JSON.parse & V8.serialize)
These encoding libraries are very convenient, because they allow in-memory objects to be saved and restored with minimal additional code. However, they also have a number of deep problems:
The encoding is often tied to a particular programming language, and reading the data in another language is very difficult.
In order to restore data in the same object types, the decoding process needs to be able to instantiate arbitrary classes. This is frequently a source of security problems.
Versioning data is often an afterthought in these libraries.
Efficiency (CPU time taken to encode or decode, and the size of the encoded structure) is also often an afterthought.
JSON, XML, and Binary Variants
Moving to standardized encodings that can be written and read by many program ming languages, JSON and XML are the obvious contenders.
JSON’s popularity is mainly due to its built-in support in web browsers (by virtue of being a subset of JavaScript) and sim plicity relative to XML.
Flaws of JSON, XML.
JSON, XML, and CSV are textual formats, and thus somewhat human-readable (although the syntax is a popular topic of debate). Besides the superficial syntactic issues, they also have some subtle problems (check img).


JSON for binary encoding
JSON is less verbose than XML, but both still use a lot of space compared to binary formats. This observation led to the development of a profusion of binary encodings for JSON (MessagePack, BSON, BJSON, UBJSON, BISON, and Smile, to name a few) and for XML (WBXML and Fast Infoset, for example). These formats have been adopted in various niches, but none of them are as widely adopted as the textual ver sions of JSON and XML.

What is a data dump?
A data dump is a complete, point-in-time export of an entire dataset or database, extracted in bulk and written to a standalone file or storage system without ongoing synchronization.
Core Characteristics
Static and Immutable: Once created, it is a frozen snapshot of the source data at a specific moment and is not modified.
Batch-Oriented: The entire volume of data is processed and written all at once, rather than streamed as continuous, real-time updates.
Format-Independent: It can be stored as raw text (e.g., CSV, SQL scripts) or converted during export into optimized binary and columnar formats (e.g., Avro, Parquet) for long-term archiving, disaster recovery, or analytics.
Thrift and Protocol Buffers
Apache Thrift and Protocol Buffers (protobuf) are binary encoding libraries that are based on the same principle.
Protocol Buffers was originally developed at Google, Thrift was originally developed at Facebook, and both were made open source in 2007–08.

How Thrift works? (BinaryProtocol)
Thrift has two different binary encoding formatsiii, called BinaryProtocol and CompactProtocol, respectively.

How Thrift works? (Compact Protocol)
The Thrift CompactProtocol encoding is semantically equivalent to BinaryProtocol, but as you can see in Figure 4-3.
One detail to note: in the schemas shown earlier, each field was marked either required or optional, but this makes no difference to how the field is encoded (nothing in the binary data indicates whether a field was required). The difference is simply that required enables a runtime check that fails if the field is not set, which can be useful for catching bugs.

Field tags and schema evolution
Each field is identified by its tag number (the numbers 1, 2, 3 in the sample schemas) and annotated with a datatype (e.g., string or integer). If a field value is not set, it is simply omitted from the encoded record.
You can change the name of a field in the schema, since the encoded data never refers to field names, but you cannot change a field’s tag, since that would make all existing encoded data invalid.
If old code (which doesn’t know about the new tag numbers you added) tries to read data written by new code, including a new field with a tag number it doesn’t recognize, it can simply ignore that field.
What about backward compatibility? As long as each field has a unique tag number, new code can always read old data, because the tag numbers still have the same meaning.
(142-143)
Changing Datatypes and schema evolution
What about changing the datatype of a field? That may be possible—check the docu mentation for details—but there is a risk that values will lose precision or get truncated.
A curious detail of Protocol Buffers is that it does not have a list or array datatype, but instead has a repeated marker for fields (which is a third option alongside required and optional). As you can see in Figure 4-4, the encoding of a repeated field is just what it says on the tin: the same field tag simply appears multiple times in the record.
Thrift has a dedicated list datatype, which is parameterized with the datatype of the list elements. This does not allow the same evolution from single-valued to multi valued as Protocol Buffers does, but it has the advantage of supporting nested lists.

Avro
Apache Avro is another binary encoding format that is interestingly different from Protocol Buffers and Thrift. It was started in 2009 as a subproject of Hadoop, as a result of Thrift not being a good fit for Hadoop’s use cases.

The writer’s schema and the reader’s schema in Avro
With Avro, when an application wants to encode some data (to write it to a file or database, to send it over the network, etc.), it encodes the data using whatever version of the schema it knows about—for example, that schema may be compiled into the application. This is known as the writer’s schema.
When an application wants to decode some data (read it from a file or database, receive it from the network, etc.), it is expecting the data to be in some schema, which is known as the reader’s schema. That is the schema the application code is relying on —code may have been generated from that schema during the application’s build process.
The key idea with Avro is that the writer’s schema and the reader’s schema don’t have to be the same—they only need to be compatible.

Schema evolution rules with Avro
With Avro, forward compatibility means that you can have a new version of the schema as writer and an old version of the schema as reader. Conversely, backward compatibility means that you can have a new version of the schema as reader and an old version as writer.
If you were to add a field that has no default value, new readers wouldn’t be able to read data written by old writers, so you would break backward compatibility.
If you were to remove a field that has no default value, old readers wouldn’t be able to read data written by new writers, so you would break forward compatibility.
In Avro: if you want to allow a field to be null, you have to use a union type. For example, union { null, long, string } field; indicates that field can be a number, or a string, or null.
(146-147)
But what is the writer’s schema?
There is an important question that we’ve glossed over so far: how does the reader know the writer’s schema with which a particular piece of data was encoded? We can’t just include the entire schema with every record, because the schema would likely be much bigger than the encoded data, making all the space savings from the binary encoding futile.
The answer depends on the context in which Avro is being used. To give a few examples

Avro schema doesn’t contain any tag numbers.
But why is this important?
What’s the problem with keeping a couple of numbers in the schema?
The difference is that Avro is friendlier to dynamically generated schemas compared to compared to Protocol Buffers and Thrift. (img ex)
This kind of dynamically generated schema simply wasn’t a design goal of Thrift or Protocol Buffers, whereas it was for Avro.

How Code generation and dynamically and static typed languages deal with Avro?
Thrift and Protocol Buffers rely on code generation: after a schema has been defined, you can generate code that implements this schema in a programming language of your choice. This is useful in statically typed languages such as Java and C++,because it allows efficient in-memory structures to be used for decoded data, and it allows type checking and autocompletion in IDEs when writing programs that access the data structures.
In dynamically typed programming languages such as JavaScript and python, there is not much point in generating code, since there is no compile-time type checker to satisfy. In the case of a dynamically generated schema (such as an Avro schema generated from a database table), code generation is an unnecessarily obstacle to getting to the data.
Avro provides optional code generation for statically typed programming languages, but it can be used just as well without any code generation.
What backward and forward compatibility matters when it comes to schemas?
In a database, the process that writes to the database encodes the data, and the process that reads from the database decodes it.
When two processes don't share memory (e.g., a web server talking to a database, or two microservices over a network), they need an agreed-upon rulebook to translate runtime objects into raw bytes (encoding) and rebuild those bytes back into objects (decoding).
Why the Schema Matters for "Compatibility"
The passage highlights forward and backward compatibility:
Backward Compatibility: A newer version of a program (with a new schema containing an added field) can still read data written by an older version.
Forward Compatibility: An older version of a program can read data written by a newer version (safely ignoring fields it doesn't recognize yet).
The schema is what makes this possible without crashing the receiving system, allowing you to deploy updates to services independently.
Schenario: The Risk of Data Loss in Read-Modify-Write Cycles with Evolving Schemas
Say you add a field to a record schema, and the newer code writes a value for that new field to the database. Subsequently, an older version of the code (which doesn’t yet know about the new field) reads the record, updates it, and writes it back. In this situation, the desirable behavior is usually for the old code to keep the new field intact, even though it couldn’t be interpreted.
The encoding formats discussed previously support such preservation of unknown fields, but sometimes you need to take care at an application level.
As illustrated in Figure 4-7. For example, if you decode a database value into model objects in the application, and later reencode those model objects, the unknown field might be lost in that translation process.

Data Outlives Code" & Schema Evolution in Databases
What does the principle "data outlives code" mean?
Why don't databases rewrite all historical data when a schema changes?
How do databases handle reading legacy rows without rewriting them?
"Data Outlives Code": New code can replace old code in minutes, but database records written five years ago persist in their original encoding alongside records written milliseconds ago.
Avoiding Migrations: Rewriting an entire dataset to match a new schema is too expensive and I/O-intensive on large volumes of data.
On-the-Fly Compatibility:
Relational DBs: Adding columns with NULL defaults modifies metadata without rewriting disk blocks; the engine fills in missing values at read time.
Document DBs (e.g., Espresso with Avro): Use schema evolution rules to translate legacy encoded records dynamically when read by newer code.
Scenario: Opportunity for Schema Normalization when putting data in archival storage.
Perhaps you take a snapshot of your database from time to time, say for backup purposes or for loading into a data warehouse. This would be a good opportunity to normalize the datab into the new schema.
In this case, the data dump will typically be encoded using the latest schema, even if the original encoding in the source database contained a mixture of schema versions from different eras. Since you’re copying the data anyway, you might as well encode the copy of the data consistently.
As the data dump is written in one go and is thereafter immutable, formats like Avro object container files are a good fit. This is also a good opportunity to encode the data in an analytics-friendly column-oriented format such as Parquet.
What is a Remote Procedure Call (RPC)?
what is its core abstraction?
and why is it fundamentally flawed?
Definition (What it is): A networking concept/technology (dating back to the 1970s) for making API requests over a network between remote services.
Core Abstraction: Location transparency — it attempts to make a remote network request look identical to calling a local in-process function or method in your code.
Historical Implementations & Limits:
EJB / Java RMI: Language-locked (Java only).
DCOM: Platform-locked (Microsoft only).
CORBA: Excessively complex; lacks forward and backward compatibility.
The Fundamental Flaw: A network request is very different from a local function call. Hiding the network creates a false illusion, ignoring real-world network issues like unpredictable latency, timeouts, unreliability, and failure modes.

What is exactly an RPC?
The term "RPC" has a broad and a narrow sense. In the broad sense, it is anything on one computer that invokes a function on another. In this sense, you are correct that HTTP is an RPC mechanism (at least, when used as such). There are even specific protocols like XML-RPC and JSON-RPC that use HTTP as their transport mechanism.
Remote Procedure Call (RPC) is a communication paradigm and distributed computing technique that allows a computer program to execute a function (or procedure) in a different address space (commonly on another computer on a shared network), without the programmer explicitly coding the underlying network communication details.
Key Components of the Definition
Procedure Call: Calling a standard function or method with inputs and receiving a return value (e.g., getUser(id)).
Remote Execution: The code inside the function physically runs on a remote server, not the local machine.
Location Transparency: The abstraction where the syntax and developer experience of making a remote call are designed to look and feel identical to calling a local in-memory function.
How do Data Encoding, Compatibility, and API Versioning work for RPC and Service Dataflow?
The Deployment Assumption:
Unlike databases, we typically assume servers update first and clients update second.
Rule: You need backward compatibility on requests (newer server can read older client's request) and forward compatibility on responses (older client can ignore new fields in newer server's response).
Compatibility Inheritance:
Thrift / gRPC (Protobuf) / Avro: Inherit schema evolution rules from their underlying encoding format.
SOAP: Uses XML schemas (evolvable, but with subtle pitfalls).
REST (JSON): Maintains compatibility by adding optional request parameters and adding new fields to response objects.
Cross-Organizational Challenge:
Service providers have no control over external clients and cannot force upgrades. Compatibility must often be maintained indefinitely, requiring providers to run multiple API versions side by side.
API Versioning Approaches:
Version number in the URL path (e.g., /v1/users).
Version header in the HTTP Accept header.
Storing the requested version on the server per API key (managed via an admin interface).
what are the several advantages a message broker has compared to direct RPC?
They are similar to RPC in that a client’s request (usually called a message) is delivered to another process with low latency. They are similar to databases in that the message is not sent via a direct network connection, but goes via an intermediary called a message broker (also called a message queue or message-oriented middleware), which stores the message temporarily
It can act as a buffer if the recipient is unavailable or overloaded, and thus improve system reliability.
It can automatically redeliver messages to a process that has crashed, and thus prevent messages from being lost.
It avoids the sender needing to know the IP address and port number of the recipient (which is particularly useful in a cloud deployment where virtual machines often come and go).
It allows one message to be sent to several recipients.
It logically decouples the sender from the recipient (the sender just publishes messages and doesn’t care who consumes them).
However, a difference compared to RPC is that message-passing communication is usually one-way: a sender normally doesn’t expect to receive a reply to its messages. It is possible for a process to send a response, but this would usually be done on a separate channel
What is The actor model?
The actor model is a programming model for concurrency in a single process. Rather than dealing directly with threads (and the associated problems of race conditions, locking, and deadlock), logic is encapsulated in actors.
Each actor typically represents one client or entity, it may have some local state (which is not shared with any other actor), and it communicates with other actors by sending and receiving asynchronous messages.
Message delivery is not guaranteed: in certain error scenarios, messages will be lost. Since each actor processes only one message at a time, it doesn’t need to worry about threads, and each actor can be scheduled independently by the framework.

where The Actor Model is used?
This model is the Distributed Actor Model (specifically utilizing Location Transparency for message passing).
It is used in distributed frameworks, systems, and industries where massive scalability, fault tolerance, and concurrency are critical.
1. Prominent Distributed Actor Frameworks
Akka / Pekko (JVM - Scala & Java): One of the most widely used distributed actor runtimes. An actor sends a message via actorRef.tell(msg, sender), and the framework automatically handles network serialization if the target actor lives on a different node in the cluster.
Microsoft Orleans (.NET / C#): Implements "Virtual Actors" (Grains) where actors are distributed automatically across a server cluster without developers managing actor placement or lifecycle.
Erlang/OTP & Elixir (BEAM): The original pioneer of this model. Node distribution is built directly into the language runtime (Node.spawn and standard PID message passing).
ProtoActor (Go, C#, Kotlin): A cross-platform, high-performance distributed actor library.
Ray (Python): Uses distributed actors under the hood to manage state and parallel execution across multi-node clusters for machine learning workloads.
How is the Actor model different from Kafka or RabbitMQ
The fundamental difference comes down to who owns the state and how communication is addressed.
Distributed actors represent active in-memory stateful entities talking directly to each other, whereas Kafka and RabbitMQ are external message brokers sitting between stateless services.
Concrete Scenario: Updating an IoT Sensor1. In a Distributed Actor Model (Akka / Orleans)
You have 100,000 IoT sensors.
You spin up 100,000 in-memory SensorActor instances across a 5-node cluster.
When a ping arrives for Sensor #42, the framework routes the message directly to the node hosting SensorActor(42).
SensorActor(42) already has its last 10 readings sitting in its own RAM, updates its local memory immediately with no database read, and checks for threshold alerts in microseconds.
2. In a Message Broker Model (Kafka / RabbitMQ)
The ping for Sensor #42 is published to a topic named sensor-readings.
The message sits in Kafka/RabbitMQ until an available worker service pulls it.
The worker doesn't know anything about Sensor #42 yet: it must fetch Sensor #42's history from a database (e.g., PostgreSQL/Redis), process the alert, write the new reading back to the database, and acknowledge the message.