Data Science quiz

5 types of text analysis

concordance, collocation, significant, named entity, sentiment

significant terms

• (term frequency - inverse document frequency)

• higher TF - IDF = more significant

• applications: machine learning, search engines, text summarizing

named entity recognition analysis

• used to identify people, places, important dates, organizations, objects

• high degree of accuracy

• used in search engines as with significant terms analysis

• summarization, job hiring, project selection

what does sentiment analysis need to be effective?

the combination of the other forms of text analysis

concordance

• refers to a list of all occurences of a particular word or phrase in a text, along with the immediate context surrounding each occurrence

• useful for pattern recognition, semantic analysis, feature extraction, information retrieval, language modeling

• useful for public opinion

collocation

• concepts in a text that cannot be expressed in a single word 

• collocations are a statistical overview of words that have a relatively high co-occurrence with a particular keyword - words close to the keyword are called collocates 

• space to either side of the keyword is the window

• important because it allow us to web of connectively between people, places, ideas, technologies, and values