经典深度学习.pptx

《经典深度学习.pptx》由会员分享，可在线阅读，更多相关《经典深度学习.pptx（136页珍藏版）》请在淘文阁 - 分享文档赚钱的网站上搜索。

1、RANLP2015,HISSAR,BULGARIADeep Learning in Industry Data Analytics Junlan FengChina Mobile Research14thNovember2016人工智能的起点:达特茅斯会议1919-20011927-20111927-20161916-2011NathanielRochester4thNovember2016人工智能的阶段1950s1980s2000sFuture 自动计算机如何为计算机编程使其能够使用语言神经网络计算规模理论自我提升抽象随机性与创造性基于规则的专家系统通用智能1234thNovember201

2、6人工智能的当前技术:存在的问题1.依赖大量的标注数据2.“窄人工智能”训练完成特定的任务3.不够稳定，安全4.不具备解释能力，模型不透明4thNovember2016人工智能的当前状态:应用4thNovember2016人工智能成为热点的原因:深度学习,强化学习大规模的,复杂的,流式的数据4thNovember2016概要1.解析白解析白宫人工智能研人工智能研发战略略计划划3.深度学深度学习及最新及最新进展展2.解析十家技解析十家技术公司的的人工智能公司的的人工智能战略略4.强化学化学习及最新及最新进展展5.深度学深度学习在企在企业数据分析中的数据分析中的应用用美国人工智能战略规划美国人工智

3、能研发战略规划4thNovember2016策略-I:在人工智能研究领域做长期研发投资 目标：.确保美国的世界领导地位.优先投资下一代人工智能技术1.推动以数据为中心的知识发现技术高效的数据清洁技术以，确保用于训练系统的数据的可信性(varascty)和正确性(appropriateness)综合考虑数据，元数据，以及人的反馈或知识异构数据，多模态数据分析和挖掘，离散数据，连续数据，时间域数据，空间域数据，时空数据，图数据小数据挖掘，强调小概率事件的重要性数据和知识尤其领域知识库的融合使用4thNovember2016策略-I:在人工智能研究领域做长期研发投资 目标：.确保美国的世界领导地位.

4、优先投资下一代人工智能技术1.推动以数据为中心的知识发现技术2.增强系统的感知能力硬件或算法能提升系统感知能力的稳健性和可靠性提升在复杂动态环境中对物体的检测，分类，辨别，识别能力提升传感器或算法对人的感知，以便系统更好地跟人的合作计算和传播感知系统的不确定性给系统以便更好的判断4thNovember2016策略-I:在人工智能研究领域做长期研发投资 目标：.确保美国的世界领导地位.优先投资下一代人工智能技术1.推动以数据为中心的知识发现技术2.增强系统的感知能力当前硬件环境和算法框架下AI的理论上限学习能力语言能力感知能力推理能力创造力计划，规划能力3.理论能力和上限4thNovember2

5、016策略-I:在人工智能研究领域做长期研发投资 目标：.确保美国的世界领导地位.优先投资下一代人工智能技术1.推动以数据为中心的知识发现技术2.增强系统的感知能力目前的AI系统均为窄人工智能，“NarrowAI”而不是“GeneralAI”GAI:灵活，多任务，有自由意志，在多认知任务中的通用能力（学习能力，语言能力，感知能力，推理能力，创造力，计划，规划能力迁移学习3.理论能力和上限4.通用AI4thNovember2016策略-I:在人工智能研究领域做长期研发投资 目标：.确保美国的世界领导地位.优先投资下一代人工智能技术1.推动以数据为中心的知识发现技术2.增强系统的感知能力多AI系统

6、的协同分布式计划和控制技术3.理论能力和上限4.通用AI5.规模化AI系统4thNovember2016策略-I:在人工智能研究领域做长期研发投资 目标：.确保美国的世界领导地位.优先投资下一代人工智能技术1.推动以数据为中心的知识发现技术2.增强系统的感知能力AI系统的自我解释能力目前AI系统的学习方法：大数据，黑盒人的学习方法：小数据，接受正规的指导规则以及各种暗示仿人的AI系统，可以做智能助理，智能辅导3.理论能力和上限4.通用AI5.规模化AI系统6.仿人类的AI技术4thNovember2016策略-I:在人工智能研究领域做长期研发投资 目标：.确保美国的世界领导地位.优先投资下一代

7、人工智能技术1.推动以数据为中心的知识发现技术2.增强系统的感知能力提升机器人的感知能力，更智能的同复杂的物理世界交互3.理论能力和上限4.通用AI5.规模化AI系统6.仿人类的AI技术7.研发实用，可靠，易用的机器人4thNovember2016策略-I:在人工智能研究领域做长期研发投资 目标：.确保美国的世界领导地位.优先投资下一代人工智能技术1.推动以数据为中心的知识发现技术2.增强系统的感知能力提升机器人的感知能力，更智能的同复杂的物理世界交互GPU：提升的内存，输入输出，时钟速度，并行能力，节能“类神经元”处理器处理基于流式，动态数据利用AI技术提升硬件能力：高性能计算，优化能源消耗

8、，增强计算性能，自我智能配置，优化数据在多核处理器和内存直接移动3.理论能力和上限4.通用AI5.规模化AI系统6.仿人类的AI技术7.研发实用，可靠，易用的机器人8.AI和硬件的相互推动4thNovember2016策略-II：开发有效的人机合作方法.不是替代人，而是跟人合作，强调人和AI系统之间的互补作用1.辅助人类的人工智能技术AI系统的设计很多是为人所用复制人类计算，决策，认知4thNovember2016策略-II：开发有效的人机合作方法.不是替代人，而是跟人合作，强调人和AI系统之间的互补作用1.辅助人类的人工智能技术2.开发增强人类的技术稳态设备穿戴设备植入设备辅助数据理解4th

9、November2016策略-II：开发有效的人机合作方法.不是替代人，而是跟人合作，强调人和AI系统之间的互补作用1.辅助人类的人工智能技术2.开发增强人类的技术数据和信息的可视化，以人可以理解的方式展现提升人和系统通信的效率3.可视化，AI-人之间的友好界面4thNovember2016策略-II：开发有效的人机合作方法.不是替代人，而是跟人合作，强调人和AI系统之间的互补作用1.辅助人类的人工智能技术2.开发增强人类的技术已成功：安静环境下的流畅的语音识未解决的：噪声环境下的识别，远场语音识别，口音，儿童语音识别，受损语音识别，语言理解，对话能力3.可视化，AI-人之间的友好界面4.研发

10、更有效的语言处理系统4thNovember2016策略 III：理解并重点关注人工智能可能带来的伦理，法律，社会方面的影响1.研究人工智能技术可能带来的伦理，法律，社会方面的影响2.期待其符合人的类规范1.AI系统从设计上需要符合人类的道德标准：公平，正义，透明，责任感4thNovember2016策略 III：理解并重点关注人工智能可能带来的伦理，法律，社会方面的影响1.研究人工智能技术可能带来的伦理，法律，社会方面的影响2.期待其符合人的类规范1.AI系统从设计上需要符合人类的道德标准：公平，正义，透明，责任感2.构建符合道德的AI技术如何将道德量化，由模糊变为精确的系统和算法设计道德通常

11、是模糊的，随文化，宗教和信仰而不同4thNovember2016策略 III：理解并重点关注人工智能可能带来的伦理，法律，社会方面的影响1.研究人工智能技术可能带来的伦理，法律，社会方面的影响2.期待其符合人的类规范1.AI系统从设计上需要符合人类的道德标准：公平，正义，透明，责任感2.构建符合道德的AI技术两层架构：由一层专门负责道德建设道德标准植入每一个工程AI步骤3.符合道德标准的AI技术的实现框架4thNovember2016策略-IV:确保人工智能系统的自身和对周围环境安全性1.在人工智能系统广泛使用之前，必须确保系统的安全性2.研究创造稳定，可依靠，可信赖，可理解，可控制的人工智能

12、系统所面临的挑战及解决办法1.提升AI系统的可解释性和透明度2.建立信任3.增强verification和validation4.自我监控，自我诊断，自我修正5.意外处理能力，防攻击能力4thNovember2016策略-V:发展人工智能技术所需的共享的数据集和共享的模拟环境1.一件重要的公益事业，同时要充分尊重企业和个人在数据中的权利和利益2.鼓励开源4thNovember2016策略-VI:评价和评测人工智能技术的标准1.开发恰当的评级策略和方法4thNovember2016策略-VII:更好的理解国家在人工智能研发方面的人力需求1.保证足够的人才资源4thNovember2016大数据和

13、人工智能数据是人工智能的来源大数据并行计算，流计算等技术是人工智能能实用化的保障人工智能是大数据，尤其复杂数据分析的主要方法4thNovember2016.Top 10 家技家技术公司的布局公司的布局4thNovember2016Google:AI-First Strategy1.Google化4亿美金购买英国伦敦大学人工智能创业公司：DeepMind2.AlphaGo3.GNC4.WaveNet5.Q-Learning2011年成立1.语音识别，合成；2.机器翻译；3.无人驾驶车.4.谷歌眼镜.5.GoogleNow.6.收购Api.ui4thNovember2016Facebook共享深度

14、学习开源代码：TorchFacbookM数字助理研究和应用：FAIR&AML4thNovember2016Apple AIAppleSiriAppleboughtEmotientandVocalIQ?4thNovember2016Partnership on AIItwill“conductresearch,recommendbestpractices,andpublishresearchunderanopenlicenseinareassuchasethics,fairnessandinclusivity;transparency,privacy,andinteroperability;co

15、llaborationbetweenpeopleandAIsystems;andthetrustworthiness,reliabilityandrobustnessofthetechnology”2016年年9月月29日日4thNovember2016Elon Musk:OpenAIPaypal,Telsla,SpaceX,SolarCity四家公司CEO，投资十个亿美金成立OpenAI4thNovember2016Microsoft小冰小娜API开放CNTK微软研究院4thNovember2016IBM语音文本图片视频Watson 计算机4thNovember2016百度4thNovemb

16、er2016国内技术巨头腾讯，阿里，讯飞在人工智能领域投入巨大4thNovember20165.深度学深度学习在企在企业数据分析中的案例数据分析中的案例4thNovember2016An example:AI in Data Analytics with Deep Learning -客户情感分析1.Introduction2.EmotionRecognitioninText3.EmotionRecognitioninSpeech4.EmotionRecognitioninConversations5.IndustrialApplicationDatasetsFeaturesMethods4t

17、hNovember2016Introduction:Interchangeable Terms 42OpinionMiningSentimentalAnalysisEmotionRecognitionPolarityDetectionReviewMining4thNovember2016Introduction:What emotions are?434thNovember2016Introduction:Problem DefinitionPositive and Negative;opinionsTarget of the opinions;Entity Related set of co

18、mponents;aspectRelated attributes;aspectOpinion holder;opinion sourceWewillonlyfocusondocumentlevelsentimentOpinionMining4thNovember2016RANLP2015,HISSAR,BULGARIAIntroduction:Text Examples6thSeptember201545athrillerwithoutalotofthrillsAnedgythrillerthatdeliversasurprisingpunchAflawedbutengrossingthri

19、llerItsunlikelywellseeabetterthrillerthisyearAneroticthrillerthatsneithertooeroticnorverythrillingeitherEmotionsareexpressedartisticallywithhelpofNegationConjunctionWordsSentimentalWords,e.g.4thNovember2016RANLP2015,HISSAR,BULGARIAIntroduction:Text Examples DSE:explicitlyexpressanopinionholdersattit

20、ude ESE:indirectlyexpresstheattitudeofthewriter6thSeptember201546Emotionsareexpressedexplicitlyandindirectly.4thNovember2016RANLP2015,HISSAR,BULGARIAIntroduction:Text Examples6thSeptember201547Emotionsareexpressedlanguagethatisoftenobscuredbysarcasm,ambiguity,andplaysonwords,allofwhichcouldbeverymis

21、leadingforbothhumansandcomputers AsharptonguedoesnotmeanyouhaveakeenmindIdontknowwhatmakesyousodumbbutitreallyworksPlease,keeptalking.Sogreat.IalwaysyawnwhenIaminterested.4thNovember2016RANLP2015,HISSAR,BULGARIAIntroduction:Speech Conversation Examples6thSeptember2015484thNovember2016RANLP2015,HISSA

22、R,BULGARIAIntroduction:Conversation Examples6thSeptember2015494thNovember2016RANLP2015,HISSAR,BULGARIATypical Approach:A Classification Task6thSeptember201550ADocumentFeatures:Ngrams(Uni,bigrams)POSTagsTermFrequencySyntacticDependencyNegationTagsSVMMaxentNaveBayesCRFRandomForestPosNeuNegSupervisedLe

23、arningPos-TagPatterns+Dictionary+MutualInfoRulesUnsupervisedLearning4thNovember2016RANLP2015,HISSAR,BULGARIATypical Approach:A Classification Task6thSeptember201551Features:Prosodic features:pitch,energy,formants,etc.Voice quality features:harsh,tense,breathy,etc.Spectral features:LPC,MFCC,LPCC,etc.

24、Teager Energy Operator(TEO)-based features:TEO-FM-var,TEO-Auto-Env,etcSVMGMMHMMDBNKNNLDACARTPosNeuNegSupervisedLearning4thNovember2016Challenges Remain1.Text-Based:CapturethecompositionaleffectswithhigheraccuracyNegatingPositivesentencesNegatingNegativesentencesConjunction:2.Speech-Based:Effectivefe

25、aturesunknown.EmotionalspeechsegmentstendtobetranscribedwithlowerASRaccuracy4thNovember2016Overview1.Introduction2.EmotionRecognitioninTextWordEmbeddingforSentimentAnalysisCNNforSentimentClassificationRNN,LSTMforsentimentClassificationPriorKnowledge+CNN/LSTMParsing+RNN3.EmotionRecognitioninSpeech4.E

26、motionRecognitioninConversations5.IndustrialApplicationHowdeeplearningcanchangethegame?4thNovember2016RANLP2015,HISSAR,BULGARIA6thSeptember201554Emotion Classification with Deep learning approaches4thNovember2016RANLP2015,HISSAR,BULGARIA1.Word Embedding as Features6thSeptember201555Representationoft

27、extisveryimportantforperformanceofmanyreal-worldapplicationsincludingemotionrecognition:Localrepresentations:N-gramsBag-of-words1-of-NcodingContinuousRepresentations:LatentSemanticAnalysisLatentDirichletAllocationDistributedRepresentations:wordembeddingTomasMikolov,“LearningRepresentationsofTextusin

28、gNeuralNetworks”,NIPsDeeplearningWorkshop2013(Bengioetal.,2006;Collobert&Weston,2008;Mnih&Hinton,2008;Turianetal.,2010;Mikolovetal.,2013a;c)4thNovember2016RANLP2015,HISSAR,BULGARIA1.Word Embedding as Features6thSeptember201556Representationoftextisveryimportantforperformanceofmanyreal-worldapplicati

29、onsincludingemotionrecognition:Localrepresentations:N-gramsBag-of-words1-of-NcodingContinuousRepresentations:LatentSemanticAnalysisLatentDirichletAllocationDistributedRepresentations:wordembeddingTomasMikolov,“LearningRepresentationsofTextusingNeuralNetworks”,NIPsDeeplearningWorkshop20134thNovember2

30、016RANLP2015,HISSAR,BULGARIAWord Embedding6thSeptember201557Skip-gramArchCBOWThehiddenlayervectoristheword-embeddingvectorforw(t)4thNovember2016Word Embedding for Sentiment DetectionIthasbeenwidelyacceptedasstandardfeaturesforNLPapplicationsincludingsentimentanalysissince2013Mikolov2013Thewordvector

31、spaceimplicitlyencodesmanylinguisticregularitiesamongwords:semanticandsyntacticExample:GooglePre-trainedwordvectorswith1000BillionwordsDoesitencodepolaritysimilarities?great0.729151bad0.719005terrific0.688912decent0.683735nice0.683609excellent0.644293fantastic0.640778better0.612073solid0.580604lousy

32、0.576420 wonderful0.572612terrible0.560204Good0.558616Top Relevant Words to“good”MostlyYes,butitdoesntseparateantonymswell4thNovember2016RANLP2015,HISSAR,BULGARIALearning Sentiment-Specific Word Embedding6thSeptember201559Tang,etal,“LearningSentimentSpecificWordEmbeddingforTwitterSentimentClassifica

33、tion”,ACL20144thNovember2016RANLP2015,HISSAR,BULGARIALearning Sentiment-Specific Word Embedding6thSeptember201560Tang,etal,“LearningSentimentSpecificWordEmbeddingforTwitterSentimentClassification”,ACL2014InSpirit,itissimilartomulti-tasklearning.Itlearnsthesamewayastheregularword-embeddingwithlossfun

34、ctionconsideringbothsemanticcontextandsentimentdistancetothetwitteremotionsymbols.4thNovember201610milliontweetsselectedbypositiveandnegativeemoticonsastrainingdataTheTwittersentimentclassificationtrackofSemEval2013Learning Sentiment-Specific Word EmbeddingTang,etal,“LearningSentimentSpecificWordEmb

35、eddingforTwitterSentimentClassification”,ACL20144thNovember2016Paragraph VectorsLeandMikolov,“DistributionalRepresentationsofSentencesandDocuments,ICML2014Paragraphvectorsaredistributionalvectorrepresentationforpiecesoftext,suchassentencesorparagraphsTheparagraphvectorsarealsoaskedtocontributetothep

36、redictiontaskofthenextwordgivenmanycontextssampledfromtheparagraph.EachparagraphcorrespondstoonecolumninDItactsasamemoryrememberingwhatismissingfromthecurrentcontext,aboutthetopicoftheparagraph4thNovember2016Paragraph Vectors Best Results on MR Data SetLeandMikolov,“DistributionalRepresentationsofSe

37、ntencesandDocuments,ICML20144thNovember2016Overview1.Introduction2.EmotionRecognitioninTextWordEmbeddingforSentimentAnalysisCNNforSentimentClassificationRNN,LSTMforsentimentClassificationPriorKnowledge+CNN/LSTMDatasetCollection3.EmotionRecognitioninSpeech4.EmotionRecognitioninConversations5.Industri

38、alApplication4thNovember2016 CNN for Sentiment Classification Ref:YoonKim.ConvolutionalNeuralNetworksforSentenceClassification.EMNLP,2014.4thNovember2016CNN for Sentiment Classification 1.Ref:YoonKim.ConvolutionalNeuralNetworksforSentenceClassification.EMNLP,2014.1.AsimpleCNNwithOneLayerofconvolutio

39、nontopofwordvectors.MotivatedbyCNNhasbeensuccessfulonmanyotherNLPtasks2.InputLayer:Wordvectorsarefrompre-trainedGoogle-Newsword2vector3.ConvLayer:Windowsize:3words,4words,5words.Eachwith100featuremap.300featuresinthepenultimatelayer4.PoolingLayer:MaxOvertimePoolingatthe5.Outputlayer:Fullyconnectedso

40、ftmaxlayer,outputdistributionoverlabels6.Regularization:Drop-outonthepenultimatelayerwithaconstrainonthel2normsoftheweightvectors7.Fine-trainembeddingvectorsduringtraining4thNovember2016Common Datasets 4thNovember2016CNN for Sentiment Classification-Results CNN-rand:Randomlyinitializeallwordembeddin

41、gs CNN-static:word2vec,keeptheembeddingsfixed CNN-nonstatic:Fine-tuningembeddingvectors4thNovember2016CNN for Sentiment Classification-Results 4thNovember2016Why it is successful?1.MultiplefiltersandmultiplefeaturemapsEmotionsareexpressedinsegments,insteadofthespanningoverthewholesentence2.Usepre-tr

42、ainedword2vecvectorsasinputfeatures.3.Embeddingwordvectorsarefurtherimprovedfornon-statictraining.Antonymsarefurtherseparatedaftertraining.4thNovember2016Resources for This work 1.SourceCode:https:/ CNN for Sentiment Kalchbrenneretal,“AConvolutionalNeuralNetworkforModelingSentences”,ACL2014Hyper Par

43、ameters in Experiments:K=4m=5,14featuremapsm=7,6featuremapsd=484thNovember2016Dynamic CNN for Sentiment Kalchbrenneretal,“AConvolutionalNeuralNetworkforModelingSentences”,ACL2014One Dimension Convolution Two Dimension Convolution48 D word vectors randomly initiated 300 D Initiated with Google word2v

44、ectorMore complicated model architecture with dynamic poolingStraight Forward 6,4 feature maps100-128 feature maps4thNovember2016JohnsonandZhang.,“EffectiveUseofWordOrderforTextCategorizationwithConvolutionalNeuralNetworks”,ACL-2015Why CNN is effective Asimpleremedyistousewordbi-gramsinadditiontouni

45、gramsIthasbeennotedthatlossofwordordercausedbybag-of-wordvectors(bowvectors)isparticularlyproblematiconsentimentclassificationComparingSVMwithTri-gramfeatureswith1,2,3windowfilterCNNTop 100 FeaturesSVMCNNUni-Grams687Bi-Grams2833Tri-Grams460SVMs cant fully take advantage of high-order ngrams4thNovemb

46、er2016Sentiment Classification Considering Features beyond Text with CNN ModelsTangetal.,“LearningSemanticRepresentationsofUsersandProductsforDocumentLevelSentimentClassification“”,ACL-20154thNovember2016Overview1.Introduction2.EmotionRecognitioninTextWordEmbeddingforSentimentAnalysisCNNforSentiment

47、ClassificationRNN,LSTMforsentimentClassificationPriorKnowledge+CNN/LSTMDatasetCollection3.EmotionRecognitioninSpeech4.EmotionRecognitioninConversations5.IndustrialApplication4thNovember2016 Recursive Neural Tensor NetworkSocheretal.,“RecursiveDeepModelsforSemanticCompositionalityoveraSentimentTreeba

48、nk”,EMNLP-2013.http:/nlp.stanford.edu/sentiment/1.TheStanfordSentimentTreebackisacorpuswithfullylabeledparsetrees2.Createdtofacilitateanalysisofthecompositionaleffectsofsentimentinlanguage3.10,662sentencesfrommoviereviews.Parsedbystanfordparser.215,154phrasesarelabeled4.AmodelcalledRecursiveNeuralTe

49、nsorNetworkswasproposed4thNovember2016 Recursive Neural Tensor Network-Distribution of sentiment values for N-gramsSocheretal.,“RecursiveDeepModelsforSemanticCompositionalityoveraSentimentTreebank”,EMNLP-2013.http:/nlp.stanford.edu/sentiment/Strongersentimentoftenbuildsupinlongerphrasesandthemajorit

50、yoftheshorterphrasesareneutral4thNovember2016 Recursive Neural Tensor Network(RNTN)Socheretal.,“RecursiveDeepModelsforSemanticCompositionalityoveraSentimentTreebank”,EMNLP-2013.http:/nlp.stanford.edu/sentiment/f=tanhVisthetensordirectlyrelateinputvectors,WistheregularRNNweightmatrix4thNovember2016Wa