(8.2.1)--8.2BasicsofNLP.pdf

《(8.2.1)--8.2BasicsofNLP.pdf》由会员分享，可在线阅读，更多相关《(8.2.1)--8.2BasicsofNLP.pdf（12页珍藏版）》请在淘文阁 - 分享文档赚钱的网站上搜索。

1、Bag of WordsA model that allows us to count all words in apiece of textCreating an occurrence matrix for the sentenceor documentBag of WordsSentences:1.Jim and Pam traveled by bus.2.The train was late.3.The flight was full.Traveling by flight isexpensive.ExampleBasic structure for a bag of wordsWord

2、s with frequenciesCombination of words Bag of wordsTF-IDFTF:Term Frequency.If a particular word appears multiple times in adocument,then it might have higher importance than the otherwords that appear fewer timesIDF:Inverse Document Frequency.If a particular word appearsmany times in a document,but

3、it is also present many times insome other documents,then maybe that word is frequent,so wecannot assign much importance to itTF-IDFSentences:1.This is the first document.2.This document is the second document.ExampleResulting Multiplication of TF-IDFTF-IDF using a logTokenizationTokenization is the

4、 process of segmenting running text into sentencesand words.In essence,its the task of cutting a text into pieces calledtokens,and at the same time throwing away certain characters,such aspunctuation.Sentences:1.This is the first document.2.This document is the second document.Thisisthefirstdocument

5、ThisisthedocumentseconddocumentStop Words RemovalSome very common words that appear to provide little orno value to the NLP objective are filtered and excludedfromthetexttobeprocessed,henceremovingwidespread and frequent terms that are not informativeabout the corresponding text.Stop words can be sa

6、fely ignored by carrying out alookup in a pre-defined list of keywords,freeing updatabase space and improving processing time.Stop Words ListStemmingStemming is used to normalize words.In English and many other languages,a single word can takemultiple forms depending upon context used.studystudiesst

7、udyingstudiedLemmatizationLemmatization has the objective of reducing a word to its baseform and grouping together different forms of the same word.bankPart of Speech TaggingPart of speech tagging is crucial for syntactic and semantic analysis.Question formationVerbContainerNounChunkingChunking means to extract meaningful phrases from unstructuredtext.Categories of phrasesPhrase structure rules