ISP610 Business Data Analytics Notes
Theory (All)
(This section covers fundamental theoretical concepts in data analytics, including statistical inference, hypothesis testing, regression analysis, and machine learning principles. It establishes a strong base for understanding more specific techniques.)
Confusion Matrix, Model Evaluation
(A confusion matrix details the performance of a classification model, breaking down results into:
True Positives (TP): Correctly predicted positive cases.
True Negatives (TN): Correctly predicted negative cases.
False Positives (FP): Incorrectly predicted positive cases (Type I error).
False Negatives (FN): Incorrectly predicted negative cases (Type II error).
Key evaluation metrics derived from the confusion matrix include:
Accuracy: (overall correctness).
Precision: (accuracy of positive predictions).
Recall: (ability to capture all positive cases).
F1-Score: (harmonic mean of precision and recall).
Model evaluation also involves techniques like cross-validation to ensure robustness and generalization, and Receiver Operating Characteristic (ROC) curves to visualize the trade-off between true positive rate and false positive rate.)
TF-IDF
(Term Frequency-Inverse Document Frequency (TF-IDF) is a numerical statistic used to evaluate the importance of a term in a document relative to a collection of documents (corpus). It is calculated as:
Where:
is the term frequency (number of times term t appears in document d).
is the inverse document frequency (inverse of the proportion of documents in corpus D that contain term t). It is calculated as , where is the total number of documents in the corpus and is the number of documents containing the term .
TF-IDF is widely used in text analysis for feature extraction, weighting terms, and improving the performance of search engines and text mining applications.)
Syntax Tree, Morpheme
(A syntax tree is a hierarchical representation of the grammatical structure of a sentence. It breaks down a sentence into its constituent parts, such as noun phrases, verb phrases, and individual words, illustrating the relationships between them.
A morpheme is the smallest unit of meaning in a language. Morphemes can be:
Free Morphemes: Can stand alone as words (e.g., 'cat', 'run').
Bound Morphemes: Must be attached to other morphemes to form words (e.g., prefixes like 'un-', suffixes like '-ing').)