Comprehensive Notes – Data Architecture, Data Sources, Data Quality & Pre-Processing

Data Architecture

• Definition

  • Composed of the following tightly-linked elements

  • Models

  • Policies

  • Rules / Standards

  • Govern what data

  • $\text{is collected}$

  • $\text{is stored}$

  • $\text{is arranged}$

  • $\text{is integrated}$

  • $\text{is put to use}$ inside data systems & across the organisation

• Position in Enterprise / Solution Architecture

  • One of the architecture domains (pillars) that collectively support enterprise or solution architecture

Constraints & Influences on Data-Architecture Design

• Enterprise requirements

• Technology drivers

• Economics constraints

• Business policies & regulatory rules

• Data-processing needs

Enterprise Requirements (business-side drivers)

• Economical & effective system expansion capability

• Acceptable performance (esp. access speed)

• High transaction reliability

• Transparent data management

• Conversion of raw data → useful information via

  • Data warehouses

  • Management decision-support processes

Architecture Techniques (organisational rules of thumb)

• Split management of

  • Transaction data

  • Master / reference data

• Separate systems of

  • Data capture

  • Data retrieval

Technology Drivers

• Arise from completed data-architecture / DB-architecture designs

• Influenced by

  • Existing integration frameworks / standards

  • Organisational economics

  • Existing site resources (e.g. previously purchased licences)

Economics

• Some theoretically optimal solutions are eliminated due to cost

• External macro-economic factors affecting choices

  • Business cycle

  • Interest rates

  • Market conditions

  • Legal considerations

Business Policies

• Internal policies

• Rules of regulatory bodies

• Professional standards

• Applicable government laws (can vary by agency)

• Together they define how the enterprise wishes to process data

Data-Processing Needs

• Accurate & reproducible high-volume transactions

• Data warehousing for MIS & possible data mining

• Repetitive periodic reporting

• Ad-hoc reporting

• Support for special initiatives (annual budget, new product development, …)

General (Three-Level) Design Approach

1. Logical level

2. Physical level

3. Implementation level
   Each successive level refines the former into technology-specific specifications.

Understanding Data Sources

Two Super-Categories

1. Primary sources (first-hand, generated specifically for purpose)

2. Secondary sources (pre-existing, generated for other purposes)

Primary Data Sources & Collection Methods

1. Observation Method

• Direct human / mechanical observation of behaviour or events

• Examples / Scenarios

  • Service-station mystery visit (researcher poses as customer)

  • In-store display evaluation (count pass-by, stop-and-look, buy decisions)

  • Super-market shelf-placement study (hidden cameras)

  • Sales-force enthusiasm test (concealed tape recorder)

• Advantages

  • No dependence on respondent willingness or ability

  • Interviewer bias reduced

  • Data generally more objective & accurate

• Disadvantages

  • Can’t observe internal states (attitudes, motives, etc.)

  • Time-consuming (must wait for events)

  • Some private activities impossible to observe

  • Usually higher cost vs. questionnaires; unproductive waiting time

2. Survey Method (Four sub-modes)

1. Telephonic interview

   • Quickest way to gather information

   • Pros: fast, clarifications possible, better response vs. mail, good info quality, lower admin cost

   • Cons: cannot use props, unsuitable for unstructured or long descriptive answers, no observation, phone access bias, reluctance to reveal personal info

2. Personal interview (face-to-face)

   • Most versatile (props + observation of non-verbal cues)

   • Pros: extra probing, flexible question order, knowledge of past/future possible, in-depth studies, data verification, reliable cause-effect detection

   • Cons: requires advanced planning/supervision, expensive, time consuming, interviewer effect, may need multiple callbacks, some respondents inaccessible

3. Mail survey

   • Questionnaire mailed → respondent completes & returns

   • Pros: reaches all groups, response can be improved by incentives

   • Cons: unsuitable for unstructured studies, costly, needs mailing list, slow turnaround, complex Q’s problematic

4. Electronic interview / electronic observation

   • Observation via electronic means (e.g. UPC scanner data)

   • Pros: independence from respondent willingness, data more objective

   • Cons: cannot capture attitudes, long-duration events, observer bias if known, costly

3. Experimental Method

• Controlled manipulation of variables; four popular experimental designs

  1. CRD – Completely Randomized Design

  • Treatments assigned at random; assumes homogeneous experimental units

  • Suitable for lab settings; field variability problematic

  1. RBD – Randomized Block Design

  • Origin: agriculture; land divided into blocks, each block gets all treatments in randomised plots; analysed via ANOVA

  1. LSD – Latin Square Design

  • Balanced $n \times n$ array where each treatment appears once per row & column (e.g. $4 \times 4$ A–D schematic)

  • Controls variation in two directions (rows & columns) giving lower error

  1. FD – Factorial Designs

  • Simultaneous testing of $\ge 2$ variables; measures interactions; relies on randomisation to infer cause–effect

Secondary Data Sources

1. Internal Sources (within the firm)

• Faster, cheaper, often more pertinent

• Key sub-sources

  • Accounting records (costs, sales, internal variables)

  • Sales-force reports (market, customer, competitor insights)

  • Internal experts (department heads)

  • Miscellaneous operational reports

2. External Sources (outside environment)

• Larger variety → harder to collect

• Government publications

  • Registrar General of India – demographic data

  • Central Statistical Organisation – national income, growth rates, Annual Survey of Industries (workers, production, value added)

  • Directorate General of Commercial Intelligence – foreign trade by region/country

  • Ministry of Commerce & Industries – wholesale price index, consumer price index

  • Planning Commission – basic statistics of Indian economy

  • Reserve Bank of India – banking, savings & investment, currency & finance reports

  • National Sample Survey Organisation – social/economic/demographic/agri statistics

  • Labour Bureau – labour statistics (skilled/unskilled, white-collar)

  • Department of Economic Affairs – economic survey: income, expenditure, investment, trade

  • State Statistical Abstracts – state-level data (commerce, education, etc.)

• Non-government publications

  • Industry / trade associations (e.g. Indian Cotton Mill Association, Chambers of Commerce)

  • Bombay Stock Exchange directories (financial metrics)

  • Export Promotion Councils, CII, SIDBI

• Syndicate (Syndicated) Services

  • Agencies collect & tabulate marketing info for multiple subscribing clients

  • Data sources: household survey, mail diary panels (purchase/media), electronic scanner panels; institutional panels (wholesalers, retailers, industrial firms)

  • Examples: ORG, IMRB

  • Advantages: specific decision-oriented data, low per-unit cost

  • Disadvantages: information not exclusive; customised research sometimes preferable

• International Organisations

  • ILO – labour, employment, wages, CPI

  • OECD – trade, industry, transport, S&T data

  • IMF – national & international FX regulations & reports

Data Quality

• DM often uses data collected for other purposes; quality problems unavoidable → must detect/correct or design tolerant algorithms

Measurement & Data Collection Issues

• Sources of imperfection

  • Human error, device limitations, process flaws

  • Missing values, spurious / duplicate objects

  • Inconsistencies (e.g. height $= 2$ m, weight $= 2$ kg)

• Error taxonomy

  • Measurement error – recorded value differs from true value; continuous errors quantified by $\text{error}=x<em>{\text{measured}}-x</em>{\text{true}}$

  • Data-collection error – omitted / extra objects or values (e.g., wrong species sampled)

  • Each can be systematic or random

Noise & Artifacts

• Noise = random component of measurement error (value distortion or false objects)

  • Common in temporal / spatial data; signal-processing methods may mitigate

• Artifact = deterministic distortion (e.g. scratch on every photo)

Precision, Bias & Accuracy

• Precision: closeness of repeated measures; usually $\sigma$ (standard deviation)

• Bias: systematic deviation; $\text{Bias}=\bar{x}-\mu<em>{\text{true}}$ when $\mu</em>{\text{true}}$ known

• Accuracy: closeness to true value; depends on both precision & bias

  • Use significant digits consistent with precision

Outliers

• Objects or attribute values far from the norm

• Can be genuine & interesting (not just noise)

Missing Values – Handling Strategies

1. Eliminate objects/attributes (if few)

2. Estimate values

   • Interpolation in smooth time series

   • Nearest-neighbour averaging (continuous) or modal value (categorical)

3. Ignore during analysis

   • Modify algorithms to compute similarity using available attributes

Inconsistent Values

• Detect (range checks, cross-field checks, external lookup) and correct if redundant info exists (check digits, code tables)

Duplicate Data (Deduplication)

• Challenges

  1. Resolve attribute conflicts when merging duplicates

  2. Avoid merging similar but distinct objects

• In some cases true duplicates are legitimate (identical rows) but must be accounted for in algorithm design

Application-Specific Issues

• Timeliness: data becomes outdated; models therefore expire

• Relevance: data must contain predictive attributes; missing critical fields ↓ accuracy

• Sampling bias: sample proportions ≠ population; e.g. survey non-responders

• Knowledge about data: good documentation (precision, feature type, units, missing-value codes, etc.) critical; otherwise analysis flawed

Data Pre-Processing Strategies & Techniques

• Goal: improve mining w.r.t time, cost, quality by selecting / altering data objects & attributes

1. Aggregation

• Combine two or more attributes or objects

• Quantitative attributes: sum or average; qualitative: omit or summarise

• Motivations

  • Smaller data sets → less memory & faster algos

  • Enables computationally expensive algorithms

  • Provides higher-level (macro) view of data

• Disadvantage: may lose interesting detail

2. Sampling

• Select subset of objects; representative if sample statistic ~ population statistic

  • Example representativeness criterion: $|\bar{x}<em>{\text{sample}}-\bar{x}</em>{\text{population}}| \le \varepsilon$

• Types

  • Random sampling

  • Without replacement (no repeats)

  • With replacement (object may be selected multiple times)

  • Progressive / Adaptive sampling: begin small, increment until properties stabilise

3. Dimensionality Reduction

• Lower number of attributes →

  • Reduced noise, easier visualisation, less time/memory, more understandable models

Feature Subset Selection

• Keep subset of original attributes; discard irrelevant/redundant

4. Feature Creation (not detailed in transcript but implicit)

• Combine, extract or derive new variables

5. Discretisation & Binarisation

• Discretisation: continuous → categorical (e.g. age $\to$ {young, middle, senior})

• Binarisation: continuous or discrete → binary indicator(s) (dummy variables)

6. Variable Transformation

• Apply functional transformation to every value (e.g. $\log, \sqrt{}, z\text{-score}$) to stabilise variance, normalise, or linearise relationships

Cross-Topic Connections & Practical Implications

• Choice of primary vs. secondary data affects data-quality workload: primary allows design of instruments to minimise noise/bias; secondary forces more cleaning

• Constraints (economics, policies) influence pre-processing choices (e.g. aggregation when storage $\$ expensive$)

• Experimental designs (CRD, RBD, LSD, FD) provide high-quality primary data with controlled precision & low bias, exemplifying "address quality at source" ideal absent in many DM contexts

• Documentation (knowledge about data) is itself an internal secondary source and feeds into preprocessing decisions (e.g. attribute types determine feasible transformations)

Ethical / Philosophical Notes

• Observational studies raise privacy concerns (hidden cameras, tape-recorders)

• Syndicated data sharing must balance low cost vs. exclusivity & competitive advantage

• Elimination or correction of outliers touches on scientific integrity: removing genuine but rare phenomena can bias results

• Timeliness & sampling bias highlight the ethical duty to acknowledge model limitations in dynamic / skewed populations