Data Science Exam
📚 THE DEFINITIVE, EXHAUSTIVE STUDY GUIDE (ALL FILES, CHINESE INCLUDED)
1. CLOUD COMPUTING (云计算)
· Definition: Distributed computing that decomposes huge data processing programs into small programs via the "cloud" network.
· Core Idea: Unifies and schedules computing resources to form a computing resource pool (计算资源池) for on-demand service.
· Core Tech: Virtualization (虚拟化) / Resource pooling (资源池化).
· Service Models (3 layers):
· IaaS (Infrastructure as a Service / 基础设施即服务): Provides VMs, storage, network.
· PaaS (Platform as a Service / 平台即服务): Provides OS, databases, dev frameworks.
· SaaS (Software as a Service / 软件即服务): Provides software (e.g., Gmail, Office 365).
· Deployment Models: Public Cloud (公有云), Private Cloud (私有云), Hybrid Cloud (混合云).
· Advantages: High scalability, low cost, high reliability.
· Flaws: Requires network, security/privacy concerns.
2. BLOCKCHAIN (区块链)
· Definition: A distributed shared ledger (账本) and database.
· Core Features: Decentralization (去中心化), Immutability (不可篡改), Traceability (可追溯), Collective Maintenance (集体维护), Openness/Transparency (公开透明).
· Cryptography: Hash Algorithm (哈希算法) and Asymmetric Encryption.
· First Application: Bitcoin (比特币) (2008, Satoshi Nakamoto).
· Stages: 1.0 Digital Currency (Bitcoin) → 2.0 Smart Contracts (智能合约) (Ethereum) → 3.0 Empowering Entities (赋能实体).
3. DATABASES (数据库)
Relational Databases (关系数据库)
· Model: 2D tables (columns = Fields (字段), rows = Records (记录)).
· ACID Properties:
· Atomicity (原子性): All or nothing (completely succeed or completely fail (失败)).
· Consistency (一致性): Always in a consistent (一致) state.
· Isolation (隔离性): Results invisible (不可见) until transaction completes.
· Durability (持久性): Results are permanent (永久).
· Disadvantages: Poor scalability, rigid data model.
NoSQL Databases (非关系型数据库)
· Meaning: Not Only SQL (不仅仅是SQL).
· Types: Key-Value (键值) (Redis), Column-Family (列族) (HBase), Document (文档) (MongoDB), Graph (图) (Neo4j).
· Advantages: High scalability (扩展性), flexibility, massive data.
· Disadvantages: Poor complex queries (复杂查询), no ACID.
NewSQL
· Combines ACID (relational) + Scalability (NoSQL).
4. STORAGE (存储介质)
· Magnetic (磁存储): Uses magnetic particles (磁颗粒). Example: HDD. Slower (慢), large capacity, cheap.
· Electrical (电存储): Uses electrons (电子). Example: SSD. Faster (快), smaller capacity, expensive.
· Why used together: Balance speed (速度) and cost. No single medium satisfies fast, large, and cheap.
· Hierarchy: Registers → Cache → Memory → Disk Cache → Disk/SSD → Tape (磁带) / Optical Disc.
· Distributed Storage (分布式存储): Sharding (分片) + Replication (复制).
5. DATA MINING (数据挖掘)
· Definition: Extracting implicit, unknown, useful knowledge from large, noisy, random data. KDD = non-trivial process.
· KNN (K-近邻): K is a critical parameter (参数). Classification by majority (大多数) of K neighbors. 1-NN = nearest neighbor (最近邻).
· Naive Bayes (朴素贝叶斯): Assumes Attribute Conditional Independence (属性条件独立性).
· Logistic Regression (逻辑回归): Binary classification (二分类). Sigmoid function maps to [0,1].
· Association Rules (关联规则):
· Support (支持度): Usefulness/frequency (有用性/频率).
· Confidence (置信度): Certainty (确定性).
· Apriori: Layer-by-layer (逐层) search. Prior property: subsets of frequent itemsets are frequent itemsets (频繁项集).
· Statistics vs. Data Mining: Statistics = Hypothesis-driven (假设驱动). Data Mining = Data-driven (数据驱动).
6. KNOWLEDGE GRAPH (知识图谱)
· Definition: Large-scale semantic (语义) network.
· Birth: 2012, Google Knowledge Graph.
· Basic Unit: Entity triple (实体三元组) — (Entity1, Relation, Entity2).
· TransE: head entity + relation = tail (尾) entity.
· Query Language: SPARQL for Graph (图) databases.
· F1-Score Calculation: F1 = 2 × (Precision × Recall) / (Precision + Recall).
7. INFORMATION SECURITY (信息安全)
· CIA Triad (CIA三元组):
· Confidentiality (保密性) (unauthorized no access).
· Integrity (完整性) (no tampering).
· Availability (可用性) (accessible when needed).
· Four Angles of Impact:
· Individual (个人): Identity theft, privacy, financial (财产) loss.
· Enterprise (企业): Reputation, financial, legal liability (责任/诉讼).
· Society (社会): Misinformation, trust (信任) erosion.
· State (国家): Critical infrastructure (基础设施), national security.
· Protection: Security tech, management, laws, awareness (意识).
8. DATA ETHICS (数据伦理)
· Belongs to: Tech (科技) ethics.
· Five Principles: Human welfare (人类福祉), Respect life (尊重生命), Fairness (公平公正), Risk control (风险控制), Openness/Transparency (公开透明).
· GDPR: 7 principles. Includes Accountability (问责制度).
· Big Data Price Discrimination (大数据杀熟): Old users pay higher (高于) prices.
· Information Cocoon (信息茧房): Guided by interests (兴趣).
· Digital Divide (数字鸿沟): Wealth (贫富) polarization.
· China's PIPL (个人信息保护法): Effective 2021.
9. BIG DATA & INTEGRATION (大数据与数据集成)
· 5Vs: Volume (start unit P), Variety (structured/semi/unstructured (非结构化)), Velocity (1-second (1秒) law), Value (low (低) density, high overall), Veracity (真实性).
· Data Integration (数据集成): Schema (模式) matching, resolves data value conflicts (数据值冲突), tuple duplication (元组重复).
· Data Reduction (数据归约): Strategies: Dimensionality reduction (维归约), Quantity reduction (数量归约), Data (数据) compression (压缩).
10. PREPROCESSING (数据预处理)
· Dirty Data (脏数据): Incomplete (不完整), Noisy (噪音), Inconsistent (不一致).
· Data Quality Measures (数据质量测度): Accuracy (准确性), Consistency (一致性), Relevance (相关性), Interpretability (可解释性), Completeness (完整性), Timeliness (时效性), Believability (可信性).
· Data Cleaning (数据清洗):
· Missing values: Ignore tuple, fill.
· Noise smoothing: Binning (分箱), Regression (回归).
· Outliers: 3σ rule (3σ原则), Boxplot (IQR), Model/Distance/Density/Cluster.
· Data Transformation (数据转换): Normalization (Min-Max (最小-最大规范化)), Discretization (离散化), Concept generalization (概念泛化).
· Data Masking (数据脱敏): Validity (有效性), Authenticity (真实性), Efficiency (高效性), Consistency (一致性), Compliance (合规性).
11. IOT & INTERNET (物联网与互联网)
· IoT Definition: Connecting objects to the internet via sensors for identification, tracking, monitoring.
· IoT Architecture (3 layers): Perception (感知层), Network (网络层), Application (应用层).
· IoT Applications: Smart Homes (智能家居), Healthcare (医疗健康), Agriculture (农业).
· Internet Data Collection (互联网数据采集): Web Crawlers (网络爬虫). Tool: Scrapy.
· Gateway Collection (网关采集): Translates data for non-network devices.
12. MATH (数学)
· Calculus (微积分): Optimization, Gradient descent (梯度下降), Backpropagation (反向传播).
· Linear Algebra (线性代数): Vectors, matrices, eigenvalues (特征值) (PCA).
· Statistics (统计学): Descriptive vs Inferential, Bayes (贝叶斯).
· Discrete Math (离散数学): Set theory (集合论), Graph theory (图论), Logic (逻辑学), Combinatorics (组合数学).
· K-value (K值): Internal correlation (KNN, K-Means).
13. CALCULATIONS (计算题) - 40 Points
· 1. Knowledge Graph (知识图谱) - 10 pts: F1-Score = 2 × (Precision × Recall) / (Precision + Recall).
· Precision = TP / (TP + FP)
· Recall = TP / (TP + FN)
· 2. Noise Processing (噪声处理) - 10 pts:
· Equal-Frequency (等频): Sort, divide into equal-size bins, mean, smooth.
· Equal-Width (等宽): Width = (Max-Min)/N.
· 3. Association Rules (关联规则) - 20 pts:
· Support (支持度) = (X and Y) / Total.
· Confidence (置信度) = (X and Y) / X.
14. SHORT ANSWER TOPICS (简答题) - 30 Points
· 1. IoT Applications (物联网运用): Smart home, Healthcare, Agriculture.
· 2. Internet (互联网): Web crawlers (Scrapy) extract unstructured data.
· 3. Data Reduction & Integration (数据归约与集成): Why (reduce volume, improve efficiency); Conflicts (schema, value, tuple).
· 4. NoSQL vs. NewSQL: NoSQL (Key-Value, Document, Graph, scalable, weak consistency). NewSQL (ACID + scalable).
· 5. Personal Ethics (个人伦理): Protect info, no misinformation, obey laws, raise awareness.
· 6. Statistics vs. Data Mining (统计学与数据挖掘区别): Statistics (hypothesis-driven), Data Mining (data-driven).
15. BONUS/ESSAY (发散思维)
· Prompt: Big Data and Large Models (AI) in a Big Data major.
· Points: AI needs big data for training; Big data uses AI for automated cleaning/analysis; Deep integration drives industry; Career relevance (data engineering + AI deployment).