Data Science Exam


📚 THE DEFINITIVE, EXHAUSTIVE STUDY GUIDE (ALL FILES, CHINESE INCLUDED)

1. CLOUD COMPUTING (云计算)

· Definition: Distributed computing that decomposes huge data processing programs into small programs via the "cloud" network.

· Core Idea: Unifies and schedules computing resources to form a computing resource pool (计算资源池) for on-demand service.

· Core Tech: Virtualization (虚拟化) / Resource pooling (资源池化).

· Service Models (3 layers):

· IaaS (Infrastructure as a Service / 基础设施即服务): Provides VMs, storage, network.

· PaaS (Platform as a Service / 平台即服务): Provides OS, databases, dev frameworks.

· SaaS (Software as a Service / 软件即服务): Provides software (e.g., Gmail, Office 365).

· Deployment Models: Public Cloud (公有云), Private Cloud (私有云), Hybrid Cloud (混合云).

· Advantages: High scalability, low cost, high reliability.

· Flaws: Requires network, security/privacy concerns.

2. BLOCKCHAIN (区块链)

· Definition: A distributed shared ledger (账本) and database.

· Core Features: Decentralization (去中心化), Immutability (不可篡改), Traceability (可追溯), Collective Maintenance (集体维护), Openness/Transparency (公开透明).

· Cryptography: Hash Algorithm (哈希算法) and Asymmetric Encryption.

· First Application: Bitcoin (比特币) (2008, Satoshi Nakamoto).

· Stages: 1.0 Digital Currency (Bitcoin) → 2.0 Smart Contracts (智能合约) (Ethereum) → 3.0 Empowering Entities (赋能实体).

3. DATABASES (数据库)

Relational Databases (关系数据库)

· Model: 2D tables (columns = Fields (字段), rows = Records (记录)).

· ACID Properties:

· Atomicity (原子性): All or nothing (completely succeed or completely fail (失败)).

· Consistency (一致性): Always in a consistent (一致) state.

· Isolation (隔离性): Results invisible (不可见) until transaction completes.

· Durability (持久性): Results are permanent (永久).

· Disadvantages: Poor scalability, rigid data model.

NoSQL Databases (非关系型数据库)

· Meaning: Not Only SQL (不仅仅是SQL).

· Types: Key-Value (键值) (Redis), Column-Family (列族) (HBase), Document (文档) (MongoDB), Graph (图) (Neo4j).

· Advantages: High scalability (扩展性), flexibility, massive data.

· Disadvantages: Poor complex queries (复杂查询), no ACID.

NewSQL

· Combines ACID (relational) + Scalability (NoSQL).

4. STORAGE (存储介质)

· Magnetic (磁存储): Uses magnetic particles (磁颗粒). Example: HDD. Slower (慢), large capacity, cheap.

· Electrical (电存储): Uses electrons (电子). Example: SSD. Faster (快), smaller capacity, expensive.

· Why used together: Balance speed (速度) and cost. No single medium satisfies fast, large, and cheap.

· Hierarchy: Registers → Cache → Memory → Disk Cache → Disk/SSD → Tape (磁带) / Optical Disc.

· Distributed Storage (分布式存储): Sharding (分片) + Replication (复制).

5. DATA MINING (数据挖掘)

· Definition: Extracting implicit, unknown, useful knowledge from large, noisy, random data. KDD = non-trivial process.

· KNN (K-近邻): K is a critical parameter (参数). Classification by majority (大多数) of K neighbors. 1-NN = nearest neighbor (最近邻).

· Naive Bayes (朴素贝叶斯): Assumes Attribute Conditional Independence (属性条件独立性).

· Logistic Regression (逻辑回归): Binary classification (二分类). Sigmoid function maps to [0,1].

· Association Rules (关联规则):

· Support (支持度): Usefulness/frequency (有用性/频率).

· Confidence (置信度): Certainty (确定性).

· Apriori: Layer-by-layer (逐层) search. Prior property: subsets of frequent itemsets are frequent itemsets (频繁项集).

· Statistics vs. Data Mining: Statistics = Hypothesis-driven (假设驱动). Data Mining = Data-driven (数据驱动).

6. KNOWLEDGE GRAPH (知识图谱)

· Definition: Large-scale semantic (语义) network.

· Birth: 2012, Google Knowledge Graph.

· Basic Unit: Entity triple (实体三元组) — (Entity1, Relation, Entity2).

· TransE: head entity + relation = tail (尾) entity.

· Query Language: SPARQL for Graph (图) databases.

· F1-Score Calculation: F1 = 2 × (Precision × Recall) / (Precision + Recall).

7. INFORMATION SECURITY (信息安全)

· CIA Triad (CIA三元组):

· Confidentiality (保密性) (unauthorized no access).

· Integrity (完整性) (no tampering).

· Availability (可用性) (accessible when needed).

· Four Angles of Impact:

· Individual (个人): Identity theft, privacy, financial (财产) loss.

· Enterprise (企业): Reputation, financial, legal liability (责任/诉讼).

· Society (社会): Misinformation, trust (信任) erosion.

· State (国家): Critical infrastructure (基础设施), national security.

· Protection: Security tech, management, laws, awareness (意识).

8. DATA ETHICS (数据伦理)

· Belongs to: Tech (科技) ethics.

· Five Principles: Human welfare (人类福祉), Respect life (尊重生命), Fairness (公平公正), Risk control (风险控制), Openness/Transparency (公开透明).

· GDPR: 7 principles. Includes Accountability (问责制度).

· Big Data Price Discrimination (大数据杀熟): Old users pay higher (高于) prices.

· Information Cocoon (信息茧房): Guided by interests (兴趣).

· Digital Divide (数字鸿沟): Wealth (贫富) polarization.

· China's PIPL (个人信息保护法): Effective 2021.

9. BIG DATA & INTEGRATION (大数据与数据集成)

· 5Vs: Volume (start unit P), Variety (structured/semi/unstructured (非结构化)), Velocity (1-second (1秒) law), Value (low (低) density, high overall), Veracity (真实性).

· Data Integration (数据集成): Schema (模式) matching, resolves data value conflicts (数据值冲突), tuple duplication (元组重复).

· Data Reduction (数据归约): Strategies: Dimensionality reduction (维归约), Quantity reduction (数量归约), Data (数据) compression (压缩).

10. PREPROCESSING (数据预处理)

· Dirty Data (脏数据): Incomplete (不完整), Noisy (噪音), Inconsistent (不一致).

· Data Quality Measures (数据质量测度): Accuracy (准确性), Consistency (一致性), Relevance (相关性), Interpretability (可解释性), Completeness (完整性), Timeliness (时效性), Believability (可信性).

· Data Cleaning (数据清洗):

· Missing values: Ignore tuple, fill.

· Noise smoothing: Binning (分箱), Regression (回归).

· Outliers: 3σ rule (3σ原则), Boxplot (IQR), Model/Distance/Density/Cluster.

· Data Transformation (数据转换): Normalization (Min-Max (最小-最大规范化)), Discretization (离散化), Concept generalization (概念泛化).

· Data Masking (数据脱敏): Validity (有效性), Authenticity (真实性), Efficiency (高效性), Consistency (一致性), Compliance (合规性).

11. IOT & INTERNET (物联网与互联网)

· IoT Definition: Connecting objects to the internet via sensors for identification, tracking, monitoring.

· IoT Architecture (3 layers): Perception (感知层), Network (网络层), Application (应用层).

· IoT Applications: Smart Homes (智能家居), Healthcare (医疗健康), Agriculture (农业).

· Internet Data Collection (互联网数据采集): Web Crawlers (网络爬虫). Tool: Scrapy.

· Gateway Collection (网关采集): Translates data for non-network devices.

12. MATH (数学)

· Calculus (微积分): Optimization, Gradient descent (梯度下降), Backpropagation (反向传播).

· Linear Algebra (线性代数): Vectors, matrices, eigenvalues (特征值) (PCA).

· Statistics (统计学): Descriptive vs Inferential, Bayes (贝叶斯).

· Discrete Math (离散数学): Set theory (集合论), Graph theory (图论), Logic (逻辑学), Combinatorics (组合数学).

· K-value (K值): Internal correlation (KNN, K-Means).

13. CALCULATIONS (计算题) - 40 Points

· 1. Knowledge Graph (知识图谱) - 10 pts: F1-Score = 2 × (Precision × Recall) / (Precision + Recall).

· Precision = TP / (TP + FP)

· Recall = TP / (TP + FN)

· 2. Noise Processing (噪声处理) - 10 pts:

· Equal-Frequency (等频): Sort, divide into equal-size bins, mean, smooth.

· Equal-Width (等宽): Width = (Max-Min)/N.

· 3. Association Rules (关联规则) - 20 pts:

· Support (支持度) = (X and Y) / Total.

· Confidence (置信度) = (X and Y) / X.

14. SHORT ANSWER TOPICS (简答题) - 30 Points

· 1. IoT Applications (物联网运用): Smart home, Healthcare, Agriculture.

· 2. Internet (互联网): Web crawlers (Scrapy) extract unstructured data.

· 3. Data Reduction & Integration (数据归约与集成): Why (reduce volume, improve efficiency); Conflicts (schema, value, tuple).

· 4. NoSQL vs. NewSQL: NoSQL (Key-Value, Document, Graph, scalable, weak consistency). NewSQL (ACID + scalable).

· 5. Personal Ethics (个人伦理): Protect info, no misinformation, obey laws, raise awareness.

· 6. Statistics vs. Data Mining (统计学与数据挖掘区别): Statistics (hypothesis-driven), Data Mining (data-driven).

15. BONUS/ESSAY (发散思维)

· Prompt: Big Data and Large Models (AI) in a Big Data major.

· Points: AI needs big data for training; Big data uses AI for automated cleaning/analysis; Deep integration drives industry; Career relevance (data engineering + AI deployment).