Fundamentals of Information Systems: Database Systems and Big Data

Introduction to Databases and Database Management Systems

• Database Definition: An organized collection of data.
• Database Management System (DBMS): A group of programs that perform the following functions:
    - Manipulate the database.
    - Provide a critical interface between the database and its users, as well as other application programs.

Data Fundamentals

• Business Necessity: Without data and the functional ability to process it, an organization could not successfully complete the majority of its business activities.
• Definition of Data: Consists of raw facts.
• Organization: Data must be organized in a meaningful way to transform it from raw facts into useful information.

Hierarchy of Data

• Bit (Binary Digit): Represents a circuit that is either on or off.
• Byte: Made up of eight bits. Each byte represents a single character.
• Character: Represented by a byte.
• Field: A name, number, or combination of characters that describes an aspect of a business object (such as a person, place, or thing) or activity.
• Record: A collection of related data fields.
• File: A collection of related records.
• Database: A collection of integrated and related files.
• The Full Hierarchy: Summarized as bits, characters, fields, records, files, and databases.

Data Entities, Attributes, and Keys

• Entity: A person, place, or thing for which data is collected, stored, and maintained.
• Attribute: A specific characteristic of an entity.
• Data Item: The specific value of an attribute.
• Primary Key: A field or set of fields that uniquely identifies a record within a database table.

The Database Approach to Data Management

• Traditional Approach: Each distinct operational system used data files dedicated specifically to that system, leading to data redundancy and isolation.
• Database Approach: Information systems share a pool of related data. Key characteristics include:
    - The ability to share data and information resources across the organization.
    - The requirement of a DBMS to manage the shared pool.

Data Modeling and Database Characteristics

• Critical Considerations for Building a Database:
    - Content: Determining what data should be collected and evaluating the cost of collection.
    - Access: Deciding what data should be provided to which users and the timing of that access.
    - Logical Structure: Determining how data should be arranged so that it makes sense to users.
    - Physical Organization: Determining where data should be physically located.
    - Archiving: Establishing how long data should be stored.
    - Security: Determining how data can be protected from unauthorized access or damage.
• Data Model: A diagram of data entities and their relationships.
• Enterprise Data Modeling: Data modeling performed at the level of the entire enterprise.
• Entity-Relationship (ER) Diagrams: Data models that use basic graphical symbols to show the organization of and relationships between data.

The Relational Database Model

• Relational Model Definition: A simple but highly useful way to organize data into collections of two-dimensional tables called relations.
• Table Structure:
    - Each row in the table represents a specific entity.
    - Each column represents an attribute of that entity.
• Domain: The range of allowable values for a specific data attribute.

Manipulating Data in Relational Databases

• Selecting: The process of eliminating rows according to certain criteria.
• Projecting: The process of eliminating columns in a table.
• Joining: The process of combining two or more tables.
• Linking: Combining two or more tables through common data attributes to form a new table containing only the unique data attributes.

Data Cleansing and Quality

• Data Cleansing (Data Cleaning/Scrubbing): The process of detecting and then correcting or deleting incomplete, incorrect, inaccurate, or irrelevant records residing in a database.
• Cost: The cost of performing data cleansing can be quite high.
• Data Validation: Distinct from data cleansing, this involves the identification of "bad data" and its rejection at the time of initial data entry.

SQL and ACID Properties

• SQL (Structured Query Language): A special-purpose programming language for accessing and manipulating data stored in a relational database.
• Standardization: In 1986, SQL was adopted by ANSI (American National Standards Institute) as the standard query language for relational databases.
• ACID Properties: SQL databases conform to ACID properties to ensure reliability:
    - Atomicity
    - Consistency
    - Isolation
    - Durability

Core DBMS Activities

• Provide a user view of the database.
• Create and modify the database.
• Store and retrieve data.
• Manipulate data and generate reports.

Creating and Modifying Databases

• Schema: A description of the entire database. It can be part of the database or a separate schema file. The DBMS references the schema to find data locations.
• Data Definition Language (DDL): A collection of instructions and commands used to define and describe data and relationships in a specific database. It allows the creator to describe data and relationships contained in the schema.
• Data Dictionary: A detailed description of all data used in the database. It can include:
    - Descriptions of data flows.
    - Information about record organization.
    - Data-processing requirements.

Storing, Retrieving, and Manipulating Data

• Data Requests: Application programs request data through the DBMS.
• Concurrency Control: A function that handles situations where two or more users or applications need to access the same record simultaneously.
• Query by Example (QBE): A visual approach to developing database queries or requests.
• Data Manipulation Language (DML): A specific language provided with a DBMS that allows users to access and modify data, perform queries, and generate reports.
• Output: A DBMS can produce documents, reports, and other outputs to help organizations achieve goals.

Database Administration Roles

• Database Administrators (DBAs): Skilled and trained IS professionals who:
    - Work with users to define data needs.
    - Apply database programming languages to craft databases.
    - Test and evaluate databases.
    - Implement changes to improve performance.
    - Assure data security from unauthorized access.
• Data Administrator: A nontechnical position responsible for defining and implementing consistent principles for data issues. Tasks include:
    - Setting data standards and definitions across all organizational databases.
    - This can be a high-level position reporting to top-level managers.

Modern Database Services and Software Interaction

• Database as a Service (DaaS):
    - The database is stored on a service provider’s servers.
    - Accessed by the client over a network, typically the Internet.
    - Database administration is handled by the service provider.
    - Example: Amazon Relational Database Service (Amazon RDS).
• Interface Roles:
    - Front-end Applications: Interact directly with people.
    - Back-end Applications: Interact with other programs or applications.
    - Example: The Library of Congress (LOC) uses a back-end application to allow Web access to its databases of books and digital media.

Big Data Concepts

• Big Data Definition: Extremely large and complex data collections that traditional data management software, hardware, and analysis processes are incapable of handling.
• Three Characteristics of Big Data (The 3 Vs):
    - Volume
    - Velocity
    - Variety
• Industries and Uses:
    - Retail: Monitoring social networks to engage brand advocates and identify adversaries.
    - Advertising/Marketing: Tracking social media comments.
    - Hospitals: Analyzing medical data and patient records.
    - Consumer Products: Gaining insight into consumer behavior via social networks.
    - Financial Services: Identifying customers for targeted and sophisticated offers.

Challenges of Big Data

• Choosing which subset of data to store.
• Determining where and how to store the data.
• Finding relevant "nuggets" of data for decision-making.
• Deriving value from relevant data.
• Identifying which data requires protection from unauthorized access.

Data Management and Governance

• Data Management: An integrated set of functions defining processes to obtain, certify as fit for use, store, secure, and process data to ensure accessibility, reliability, and timeliness.
• Data Governance: Defines roles, responsibilities, and processes to ensure data can be trusted and used by the entire organization.
• Drivers for Data Management:
    - Meeting external regulations to manage risk (e.g., financial misstatement).
    - Avoiding inadvertent release of sensitive data.
    - Ensuring high data quality for key decisions.
• The Governance Team: Recommended to include executives, project managers, line-of-business managers, and data stewards.
• Data Steward: An individual responsible for the management of critical data elements.
• Data Lifecycle Management (DLM): A policy-based approach to managing the flow of an enterprise’s data.

Data Warehouses, Data Marts, and Data Lakes

• Data Warehouse: A large database that collects business information from many sources in the enterprise to support management decision-making.
• ETL Process: Consists of three steps:
    - Extract
    - Transform
    - Load
• Data Mart: A subset of a data warehouse used by small-to-medium businesses or specific departments. It may contain more detailed data for a specific area than the main warehouse.
• Data Lake (Enterprise Data Hub): A "store everything" approach saving data in its raw, unaltered form. Data is only extracted and structured when accessed for a specific analysis.

NoSQL Databases and Distributed Frameworks

• NoSQL Database: Stores and retrieves data modeled using means other than two-dimensional tabular relations.
    - Advantages: Ability to spread data over multiple servers (sharding), no predefined schema requirement, flexible data structures, and improved access speed/redundancy.
• Hadoop: An open-source software framework for storing and processing extremely large data sets. It has two components:
    - MapReduce: The data processing component.
    - Hadoop Distributed File System (HDFS): The distributed file system component.

In-Memory Databases

• In-Memory Database (IMDB): A DBMS that stores the entire database in random access memory (RAM).
• Benefits: Accesses data much faster than secondary storage; enables analysis of big data.
• Performance: Performs best on multiple multicore CPUs.