Chapter 5: Data & Knowledge Management

What are the difficulties of managing data?

• Data volume: The amount of data produced increases exponentially over time.

• Data scatter: Data is collected by many individuals using different devices, formats, and systems across an organization.

• Multiple sources: Data comes from internal systems, external partners, social media, sensors, and public databases.

• Data rot: Physical media (like magnetic tapes) degrade over time, or the hardware/software required to read old formats becomes obsolete.

• Data security, quality, and integrity: Keeping data safe from unauthorized access while ensuring it remains accurate, complete, and reliable.

• Legal and regulatory compliance: Rules regarding data privacy and record retention change constantly.

What are data properties?

Data properties refer to the basic characteristics that define data in information systems:

• Integrity: Accuracy, consistency, and trustworthiness of data across its lifecycle.

• Granularity: The level of detail of the data (e.g., individual sale details vs. annual summarized revenue).

• Timeliness: How up-to-date or relevant the data is for decision-making.

• Accessibility: How easily authorized users can retrieve the data when needed.

What's data governance, master data management, master data?

• Data Governance: An overarching approach to managing information across an entire organization. It establishes clear policies, processes, roles, and responsibilities for data quality, security, and usage.

• Master Data Management (MDM): A core process within data governance that links and consolidates all enterprise data into a single, authoritative source (a "single version of the truth").

• Master Data: A set of core data entities (e.g., customer names, product IDs, employee numbers, supplier details) that span enterprise information systems and provide context for transactional data.

Data file

A collection of logically related records stored as a single unit on a computer system

Redundancy

Storing the same data in multiple places, which wastes storage space and causes inconsistencies

Isolation

When applications cannot access data trapped inside other applications or formats

Inconsistency

Occurs when different copies of the same data do not match (e.g., a customer’s address is updated in billing but not shipping)

Security

Protecting data against unauthorized access, modification, or destruction.

Integrity

Ensuring data meets specified constraints (e.g., a social security number field must contain exactly 9 digits).

Independence

Applications and data are separated so that changes to data storage structure do not require rewriting application programs.

What’s a bit, field, record, file, table, database?

These represent the data hierarchy from smallest to largest:

• Bit (Binary Digit): The smallest unit of data a computer can process (00 or 11).

• Byte: A group of 8 bits representing a single character (letter, number, or symbol).

• Field: A logical grouping of characters that forms a word, number, or code (e.g., a customer's First_Name).

• Record: A logical group of related fields describing an entity (e.g., all information about one student).

• File / Table: A collection of logically related records (e.g., a list of all enrolled students).

• Database: A group of logically related files or tables stored together.

What’s the most popular database type?

The Relational Database Management System (RDBMS). It organizes data into two-dimensional tables (rows and columns) linked together through shared key fields.

What are the three types of databases discussed in class?

1. Relational Databases: Organizes data into structured tables with rows and columns linked by relationships.

2. Object oriented databases: Stores data and operations as “objects” (useful for multimedia, complex graphics, and engineering applications)

3. Multidimensional/ data warehouse database: optimized for fast querying and analysis across multiple dimensions (e.g., sales by product, region, and time)

Primary Key

A field (or set of fields) that uniquely identifies each record in a database table so it cannot be duplicated (e.g., Student_ID, or SSN)

Entity

A person, place, thing, or event about which an organization maintains information (E.g., Customer, Order, Product)

Foreign Key

A field in one table that references the primary key of another table, creating a logical link/relationship between them.

Big Data and the characteristics

Big Data: Large, complex sets of data that cannot be efficiently managed or analyzed using traditional database software

Characteristics (The 3 Vs)

1. Volume: Enormous quantities of data generated constantly

2. Velocity: The rapid rate at which new data is generated and must be processed.

3. Variety: Diverse formats ranging from structured numbers to unstructured text, video, audio, and sensor feeds

Additional Vs often include: Veracity (truthfulness/quality and Value)

How can organizations put big data to use?

• Customize marketing: Analyzing buying behavior to send targeted advertisements and recommendations

• Operational efficiency: optimizing supply chains and predicting equipment failures before they happen

• Risk management: Detecting financial fraud in real time

• Product development: Analyzing customer feedback to design better products and services

Why do we have federal regulations when dealing with data?

Federal regulations exist to protect consumer privacy, ensure financial transparency, prevent corporate fraud, safeguard personal identity data, and maintain public trust in digital transactions.

What is the Sarbanes-Oxley Act of 2002?

A US federal law passed to protect investors from corporate fraud following major accounting scandals (such as Enron and WorldCom). It requires public companies to maintain strict internal controls, document financial reporting processes, and preserve financial records accurately.

How can big data be used in functional areas of an organization?

• HR: Predicting employee turnover and identifying qualified job candidates.

• Finance: Detecting fraudulent transactions and automating credit risk scoring.

• Marketing: Creating personalized customer segments and measuring campaign performance.

• Operations: Monitoring machine health using IoT sensors to reduce downtime.

Data Warehouse

A repository of historical data collected from multiple operational systems across the organization, organized for reporting and analysis.

Data Mart

A smaller, lower-cost version of a data warehouse designed for a specific business unit or department (e.g., a Marketing Data Mart).

How is data organized?

Data is organized hierarchically from lowest to highest: Bits →→ Bytes →→ Fields →→ Records →→Tables/Files →→ Databases.

What are examples of data sources?

• Internal: Transaction processing systems (TPS), enterprise databases, sales receipts, customer feedback surveys.

• External: Government reports, economic statistics, social media activity, weather reports, industry research.

• Devices/Sensors: Smart meters, RFID tags, security cameras, GPS units.

Data Governance

sets organizational policies for managing data quality and security

Master data

Core set of reference data shared across systems

Knowledge management

A process that helps organizations manipulate important knowledge that is part of the organization’s memory, usually in an unstructured format.

Intellectual capital

The collective knowledge skills, experience, and information assets owned by an organization

Explicit Knowledge

Objective, rational, and technical knowledge that has been documented and codified in manuals, reports, and procedures.

Tacit Knowledge

Subjective or experiential learning stored in an individual’s mind (e.g., insights, intuitions, trade secrets) that is difficult to document.

Normalization

A database design technique used to organize fields and tables in a relational database to minimize data redundancy and improve data integrity.

What are the main relationships used in databases?

• One-to-One (1:1): One record in Table A is associated with exactly one record in Table B (e.g., an Employee has one Parking Permit).

• One-to-Many (1:M): One record in Table A is linked to multiple records in Table B (e.g., one Customer can place many Orders).

• Many-to-Many (M:N): Multiple records in Table A link to multiple records in Table B (e.g., many Students take many Courses). Requires a junction/associative table in relational databases.