(8.2.1)--8.2BasicsofNLP.pdf

Bag of WordsA model that allows us to count all words in apiece of textCreating an occurrence matrix for the sentenceor documentBag of WordsSentences:1.Jim and Pam traveled by bus.2.The train was late.3.The flight was full.Traveling by flight isexpensive.ExampleBasic structure for a bag of wordsWords with frequenciesCombination of words Bag of wordsTF-IDFTF:Term Frequency.If a particular word appears multiple times in adocument,then it might have higher importance than the otherwords that appear fewer timesIDF:Inverse Document Frequency.If a particular word appearsmany times in a document,but it is also present many times insome other documents,then maybe that word is frequent,so wecannot assign much importance to itTF-IDFSentences:1.This is the first document.2.This document is the second document.ExampleResulting Multiplication of TF-IDFTF-IDF using a logTokenizationTokenization is the process of segmenting running text into sentencesand words.In essence,its the task of cutting a text into pieces calledtokens,and at the same time throwing away certain characters,such aspunctuation.Sentences:1.This is the first document.2.This document is the second document.ThisisthefirstdocumentThisisthedocumentseconddocumentStop Words RemovalSome very common words that appear to provide little orno value to the NLP objective are filtered and excludedfromthetexttobeprocessed,henceremovingwidespread and frequent terms that are not informativeabout the corresponding text.Stop words can be safely ignored by carrying out alookup in a pre-defined list of keywords,freeing updatabase space and improving processing time.Stop Words ListStemmingStemming is used to normalize words.In English and many other languages,a single word can takemultiple forms depending upon context used.studystudiesstudyingstudiedLemmatizationLemmatization has the objective of reducing a word to its baseform and grouping together different forms of the same word.bankPart of Speech TaggingPart of speech tagging is crucial for syntactic and semantic analysis.Question formationVerbContainerNounChunkingChunking means to extract meaningful phrases from unstructuredtext.Categories of phrasesPhrase structure rules