TXSA & Text Analytics: Comprehensive Study Notes

Text Analytics and Text Mining are broad terms for technologies that analyze and process semi-structured and unstructured text data. Their primary goal is to extract novel, useful patterns, derive High Quality Information, and generate Actionable Knowledge from text.

  • Crucial statistic: Approximately 85%−90%85\%-90\% of data is unstructured; tapping into this is essential for competitive advantage.

  • Text Mining typically operates on unstructured data to impose structure, while Text Analytics often deals with structured representations derived from text. The workflow involves first turning unstructured text into structured data (Text Mining), then analyzing the structured data (Text Analytics).

  • The field encompasses:

    • Natural Language Processing (NLP) and text representation.

    • Information Retrieval (IR): finding relevant information from large text corpora.

    • Information Extraction (IE): identifying and extracting facts and relationships.

    • Document Classification and Clustering.

    • Topic Mining and Sentiment Analysis.

    • Web Mining.

    • Text-based Prediction.

  • Techniques include tokenization, POS tagging, lemmatization, stemming, inverted indexes, document similarity, word clouds, and topic modeling.

  • Text analytics converts free-form, qualitative text into quantitative signals for decision making, enabling discovery beyond exact keyword matches and providing integrated analytics across data types.

  • Common tools and languages include Python, R, RapidMiner, Azure, Power BI, and KNIME.