Semester Overview

The topics for the semester include storage systems, file systems, and big data software, with next week's focus on Firebase (a NoSQL database). Storage systems will primarily cover SSDs but will also explore hard disk drives. A key concept is the block, where hard disk drives use blocks (typically 4KB) and HDFS uses logical blocks (e.g., 128MB). Data is read and written in entire blocks, with workload types being either sequential (data read one after another) or random (data read from different locations), sequential being faster. SSDs are generally faster than hard disk drives but have trade-offs. File systems, such as macOS and Windows, are located above storage systems, and files are stored as blocks in the storage system. A key concept here is the I-node (Index Node), where every file has a user-defined name and an internal integer name (I-number). The I-number locates metadata, including block locations. There's a distinction between data (file content) and metadata (modification time, file name, file size, location of blocks). HDFS (Hadoop Distributed File System) uses a data structure to store metadata, including the location of blocks, potentially on different machines, with dedicated machines serving metadata. The key takeaway is the difference between metadata and data, with different file systems organizing information differently. File formats include JSON (curly braces for objects, square brackets for arrays, can be nested) and XML (still used, e.g., Android apps, Java JAR files). Relational databases (RDBMS) involve relations meaning tables, such as MySQL, which stores tables with rows and columns. Data modeling includes the ER model and relational data model, with a focus on removing redundancy during database design. The query language is SQL (Structured data in tables), where every row has the same number of columns and a special value (NULL) if a value is missing. The physical data model involves how records are packed into blocks (e.g., 4KB), with B+ trees as an index structure where each node stores hundreds of keys in a physical block, used in NoSQL databases like Cassandra. Query execution focuses on how to join two tables using methods like loop, sorting-based, and hashing-based joins. The discussion on NoSQL vs. Relational Databases is based on ACID properties vs. CAP theory, with trade-offs between availability and consistency. ER diagrams, where E means entity set and R means relationship, are transformed into relational data models. SQL is declarative, defining needed data, data source, and conditions, and its execution plan involves common functions like filtering, joining, and projection. Big Data and NoSQL include Firebase, MongoDB (a NoSQL database with sharding to partition data across machines, designed with scale in mind), DynamoDB (using partition keys), and Cassandra (with similarities). MapReduce involves sorting, external sorting, and hashing to help understand its internals. Cassandra data writing involves writing to a commit log file (append-only, sequential write), data goes to an in-memory data structure (memtable), which is later flushed to storage, and compaction combines multiple versions of tables via external sorting. The storage system characteristics influence database system performance. Python functions include map (e.g., doubling values in a list using different methods like list comprehension, generator, and lambda function), filter (e.g., finding even numbers in a list using a Boolean value), and reduce (doing data reduction with the need for a function tool library, e.g., summation of a list using an accumulator, with two versions: with and without initial value). Setting up an EC2 instance involves creating an AWS account, signing in to the console, going to services, looking for and clicking compute, and then clicking EC2. Creating an EC2 involves navigating to instances on