ISP610 Business Data Analytics Notes

Theory (All)

(This section covers fundamental theoretical concepts in data analytics, including statistical inference, hypothesis testing, regression analysis, and machine learning principles. It establishes a strong base for understanding more specific techniques.)

Confusion Matrix, Model Evaluation

(A confusion matrix details the performance of a classification model, breaking down results into:

  • True Positives (TP): Correctly predicted positive cases.

  • True Negatives (TN): Correctly predicted negative cases.

  • False Positives (FP): Incorrectly predicted positive cases (Type I error).

  • False Negatives (FN): Incorrectly predicted negative cases (Type II error).

Key evaluation metrics derived from the confusion matrix include:

  • Accuracy: (TP+TNTP+TN+FP+FN)(\frac{TP + TN}{TP + TN + FP + FN}) (overall correctness).

  • Precision: (TPTP+FP)(\frac{TP}{TP + FP}) (accuracy of positive predictions).

  • Recall: (TPTP+FN)(\frac{TP}{TP + FN}) (ability to capture all positive cases).

  • F1-Score: (2"."Precision"."RecallPrecision+Recall)(\frac{2 "." Precision "." Recall}{Precision + Recall}) (harmonic mean of precision and recall).

Model evaluation also involves techniques like cross-validation to ensure robustness and generalization, and Receiver Operating Characteristic (ROC) curves to visualize the trade-off between true positive rate and false positive rate.)

TF-IDF

(Term Frequency-Inverse Document Frequency (TF-IDF) is a numerical statistic used to evaluate the importance of a term in a document relative to a collection of documents (corpus). It is calculated as:

TFIDF(t,d,D)=TF(t,d)⋅IDF(t,D)TFIDF(t,d,D) = TF(t,d) \cdot IDF(t,D)

Where:

  • TF(t,d)TF(t,d) is the term frequency (number of times term t appears in document d).

  • IDF(t,D)IDF(t,D) is the inverse document frequency (inverse of the proportion of documents in corpus D that contain term t). It is calculated as IDF(t,D)=log(Ndf(t)+1)IDF(t, D) = log(\frac{N}{df(t)} + 1), where NN is the total number of documents in the corpus and df(t)df(t) is the number of documents containing the term tt.

TF-IDF is widely used in text analysis for feature extraction, weighting terms, and improving the performance of search engines and text mining applications.)

Syntax Tree, Morpheme

(A syntax tree is a hierarchical representation of the grammatical structure of a sentence. It breaks down a sentence into its constituent parts, such as noun phrases, verb phrases, and individual words, illustrating the relationships between them.

A morpheme is the smallest unit of meaning in a language. Morphemes can be:

  • Free Morphemes: Can stand alone as words (e.g., 'cat', 'run').

  • Bound Morphemes: Must be attached to other morphemes to form words (e.g., prefixes like 'un-', suffixes like '-ing').)