经典深度学习.pptx

RANLP2015,HISSAR,BULGARIADeep Learning in Industry Data Analytics Junlan FengChina Mobile Research14thNovember2016人工智能的起点:达特茅斯会议1919-20011927-20111927-20161916-2011NathanielRochester4thNovember2016人工智能的阶段1950s1980s2000sFuture 自动计算机如何为计算机编程使其能够使用语言神经网络计算规模理论自我提升抽象随机性与创造性基于规则的专家系统通用智能1234thNovember2016人工智能的当前技术:存在的问题1.依赖大量的标注数据2.“窄人工智能”训练完成特定的任务3.不够稳定，安全4.不具备解释能力，模型不透明4thNovember2016人工智能的当前状态:应用4thNovember2016人工智能成为热点的原因:深度学习,强化学习大规模的,复杂的,流式的数据4thNovember2016概要1.解析白解析白宫人工智能研人工智能研发战略略计划划3.深度学深度学习及最新及最新进展展2.解析十家技解析十家技术公司的的人工智能公司的的人工智能战略略4.强化学化学习及最新及最新进展展5.深度学深度学习在企在企业数据分析中的数据分析中的应用用美国人工智能战略规划美国人工智能研发战略规划4thNovember2016策略-I:在人工智能研究领域做长期研发投资 目标：.确保美国的世界领导地位.优先投资下一代人工智能技术1.推动以数据为中心的知识发现技术高效的数据清洁技术以，确保用于训练系统的数据的可信性(varascty)和正确性(appropriateness)综合考虑数据，元数据，以及人的反馈或知识异构数据，多模态数据分析和挖掘，离散数据，连续数据，时间域数据，空间域数据，时空数据，图数据小数据挖掘，强调小概率事件的重要性数据和知识尤其领域知识库的融合使用4thNovember2016策略-I:在人工智能研究领域做长期研发投资 目标：.确保美国的世界领导地位.优先投资下一代人工智能技术1.推动以数据为中心的知识发现技术2.增强系统的感知能力硬件或算法能提升系统感知能力的稳健性和可靠性提升在复杂动态环境中对物体的检测，分类，辨别，识别能力提升传感器或算法对人的感知，以便系统更好地跟人的合作计算和传播感知系统的不确定性给系统以便更好的判断4thNovember2016策略-I:在人工智能研究领域做长期研发投资 目标：.确保美国的世界领导地位.优先投资下一代人工智能技术1.推动以数据为中心的知识发现技术2.增强系统的感知能力当前硬件环境和算法框架下AI的理论上限学习能力语言能力感知能力推理能力创造力计划，规划能力3.理论能力和上限4thNovember2016策略-I:在人工智能研究领域做长期研发投资 目标：.确保美国的世界领导地位.优先投资下一代人工智能技术1.推动以数据为中心的知识发现技术2.增强系统的感知能力目前的AI系统均为窄人工智能，“NarrowAI”而不是“GeneralAI”GAI:灵活，多任务，有自由意志，在多认知任务中的通用能力（学习能力，语言能力，感知能力，推理能力，创造力，计划，规划能力迁移学习3.理论能力和上限4.通用AI4thNovember2016策略-I:在人工智能研究领域做长期研发投资 目标：.确保美国的世界领导地位.优先投资下一代人工智能技术1.推动以数据为中心的知识发现技术2.增强系统的感知能力多AI系统的协同分布式计划和控制技术3.理论能力和上限4.通用AI5.规模化AI系统4thNovember2016策略-I:在人工智能研究领域做长期研发投资 目标：.确保美国的世界领导地位.优先投资下一代人工智能技术1.推动以数据为中心的知识发现技术2.增强系统的感知能力AI系统的自我解释能力目前AI系统的学习方法：大数据，黑盒人的学习方法：小数据，接受正规的指导规则以及各种暗示仿人的AI系统，可以做智能助理，智能辅导3.理论能力和上限4.通用AI5.规模化AI系统6.仿人类的AI技术4thNovember2016策略-I:在人工智能研究领域做长期研发投资 目标：.确保美国的世界领导地位.优先投资下一代人工智能技术1.推动以数据为中心的知识发现技术2.增强系统的感知能力提升机器人的感知能力，更智能的同复杂的物理世界交互3.理论能力和上限4.通用AI5.规模化AI系统6.仿人类的AI技术7.研发实用，可靠，易用的机器人4thNovember2016策略-I:在人工智能研究领域做长期研发投资 目标：.确保美国的世界领导地位.优先投资下一代人工智能技术1.推动以数据为中心的知识发现技术2.增强系统的感知能力提升机器人的感知能力，更智能的同复杂的物理世界交互GPU：提升的内存，输入输出，时钟速度，并行能力，节能“类神经元”处理器处理基于流式，动态数据利用AI技术提升硬件能力：高性能计算，优化能源消耗，增强计算性能，自我智能配置，优化数据在多核处理器和内存直接移动3.理论能力和上限4.通用AI5.规模化AI系统6.仿人类的AI技术7.研发实用，可靠，易用的机器人8.AI和硬件的相互推动4thNovember2016策略-II：开发有效的人机合作方法.不是替代人，而是跟人合作，强调人和AI系统之间的互补作用1.辅助人类的人工智能技术AI系统的设计很多是为人所用复制人类计算，决策，认知4thNovember2016策略-II：开发有效的人机合作方法.不是替代人，而是跟人合作，强调人和AI系统之间的互补作用1.辅助人类的人工智能技术2.开发增强人类的技术稳态设备穿戴设备植入设备辅助数据理解4thNovember2016策略-II：开发有效的人机合作方法.不是替代人，而是跟人合作，强调人和AI系统之间的互补作用1.辅助人类的人工智能技术2.开发增强人类的技术数据和信息的可视化，以人可以理解的方式展现提升人和系统通信的效率3.可视化，AI-人之间的友好界面4thNovember2016策略-II：开发有效的人机合作方法.不是替代人，而是跟人合作，强调人和AI系统之间的互补作用1.辅助人类的人工智能技术2.开发增强人类的技术已成功：安静环境下的流畅的语音识未解决的：噪声环境下的识别，远场语音识别，口音，儿童语音识别，受损语音识别，语言理解，对话能力3.可视化，AI-人之间的友好界面4.研发更有效的语言处理系统4thNovember2016策略 III：理解并重点关注人工智能可能带来的伦理，法律，社会方面的影响1.研究人工智能技术可能带来的伦理，法律，社会方面的影响2.期待其符合人的类规范1.AI系统从设计上需要符合人类的道德标准：公平，正义，透明，责任感4thNovember2016策略 III：理解并重点关注人工智能可能带来的伦理，法律，社会方面的影响1.研究人工智能技术可能带来的伦理，法律，社会方面的影响2.期待其符合人的类规范1.AI系统从设计上需要符合人类的道德标准：公平，正义，透明，责任感2.构建符合道德的AI技术如何将道德量化，由模糊变为精确的系统和算法设计道德通常是模糊的，随文化，宗教和信仰而不同4thNovember2016策略 III：理解并重点关注人工智能可能带来的伦理，法律，社会方面的影响1.研究人工智能技术可能带来的伦理，法律，社会方面的影响2.期待其符合人的类规范1.AI系统从设计上需要符合人类的道德标准：公平，正义，透明，责任感2.构建符合道德的AI技术两层架构：由一层专门负责道德建设道德标准植入每一个工程AI步骤3.符合道德标准的AI技术的实现框架4thNovember2016策略-IV:确保人工智能系统的自身和对周围环境安全性1.在人工智能系统广泛使用之前，必须确保系统的安全性2.研究创造稳定，可依靠，可信赖，可理解，可控制的人工智能系统所面临的挑战及解决办法1.提升AI系统的可解释性和透明度2.建立信任3.增强verification和validation4.自我监控，自我诊断，自我修正5.意外处理能力，防攻击能力4thNovember2016策略-V:发展人工智能技术所需的共享的数据集和共享的模拟环境1.一件重要的公益事业，同时要充分尊重企业和个人在数据中的权利和利益2.鼓励开源4thNovember2016策略-VI:评价和评测人工智能技术的标准1.开发恰当的评级策略和方法4thNovember2016策略-VII:更好的理解国家在人工智能研发方面的人力需求1.保证足够的人才资源4thNovember2016大数据和人工智能数据是人工智能的来源大数据并行计算，流计算等技术是人工智能能实用化的保障人工智能是大数据，尤其复杂数据分析的主要方法4thNovember2016.Top 10 家技家技术公司的布局公司的布局4thNovember2016Google:AI-First Strategy1.Google化4亿美金购买英国伦敦大学人工智能创业公司：DeepMind2.AlphaGo3.GNC4.WaveNet5.Q-Learning2011年成立1.语音识别，合成；2.机器翻译；3.无人驾驶车.4.谷歌眼镜.5.GoogleNow.6.收购Api.ui4thNovember2016Facebook共享深度学习开源代码：TorchFacbookM数字助理研究和应用：FAIR&AML4thNovember2016Apple AIAppleSiriAppleboughtEmotientandVocalIQ?4thNovember2016Partnership on AIItwill“conductresearch,recommendbestpractices,andpublishresearchunderanopenlicenseinareassuchasethics,fairnessandinclusivity;transparency,privacy,andinteroperability;collaborationbetweenpeopleandAIsystems;andthetrustworthiness,reliabilityandrobustnessofthetechnology”2016年年9月月29日日4thNovember2016Elon Musk:OpenAIPaypal,Telsla,SpaceX,SolarCity四家公司CEO，投资十个亿美金成立OpenAI4thNovember2016Microsoft小冰小娜API开放CNTK微软研究院4thNovember2016IBM语音文本图片视频Watson 计算机4thNovember2016百度4thNovember2016国内技术巨头腾讯，阿里，讯飞在人工智能领域投入巨大4thNovember20165.深度学深度学习在企在企业数据分析中的案例数据分析中的案例4thNovember2016An example:AI in Data Analytics with Deep Learning -客户情感分析1.Introduction2.EmotionRecognitioninText3.EmotionRecognitioninSpeech4.EmotionRecognitioninConversations5.IndustrialApplicationDatasetsFeaturesMethods4thNovember2016Introduction:Interchangeable Terms 42OpinionMiningSentimentalAnalysisEmotionRecognitionPolarityDetectionReviewMining4thNovember2016Introduction:What emotions are?434thNovember2016Introduction:Problem DefinitionPositive and Negative;opinionsTarget of the opinions;Entity Related set of components;aspectRelated attributes;aspectOpinion holder;opinion sourceWewillonlyfocusondocumentlevelsentimentOpinionMining4thNovember2016RANLP2015,HISSAR,BULGARIAIntroduction:Text Examples6thSeptember201545athrillerwithoutalotofthrillsAnedgythrillerthatdeliversasurprisingpunchAflawedbutengrossingthrillerItsunlikelywellseeabetterthrillerthisyearAneroticthrillerthatsneithertooeroticnorverythrillingeitherEmotionsareexpressedartisticallywithhelpofNegationConjunctionWordsSentimentalWords,e.g.4thNovember2016RANLP2015,HISSAR,BULGARIAIntroduction:Text Examples DSE:explicitlyexpressanopinionholdersattitude ESE:indirectlyexpresstheattitudeofthewriter6thSeptember201546Emotionsareexpressedexplicitlyandindirectly.4thNovember2016RANLP2015,HISSAR,BULGARIAIntroduction:Text Examples6thSeptember201547Emotionsareexpressedlanguagethatisoftenobscuredbysarcasm,ambiguity,andplaysonwords,allofwhichcouldbeverymisleadingforbothhumansandcomputers AsharptonguedoesnotmeanyouhaveakeenmindIdontknowwhatmakesyousodumbbutitreallyworksPlease,keeptalking.Sogreat.IalwaysyawnwhenIaminterested.4thNovember2016RANLP2015,HISSAR,BULGARIAIntroduction:Speech Conversation Examples6thSeptember2015484thNovember2016RANLP2015,HISSAR,BULGARIAIntroduction:Conversation Examples6thSeptember2015494thNovember2016RANLP2015,HISSAR,BULGARIATypical Approach:A Classification Task6thSeptember201550ADocumentFeatures:Ngrams(Uni,bigrams)POSTagsTermFrequencySyntacticDependencyNegationTagsSVMMaxentNaveBayesCRFRandomForestPosNeuNegSupervisedLearningPos-TagPatterns+Dictionary+MutualInfoRulesUnsupervisedLearning4thNovember2016RANLP2015,HISSAR,BULGARIATypical Approach:A Classification Task6thSeptember201551Features:Prosodic features:pitch,energy,formants,etc.Voice quality features:harsh,tense,breathy,etc.Spectral features:LPC,MFCC,LPCC,etc.Teager Energy Operator(TEO)-based features:TEO-FM-var,TEO-Auto-Env,etcSVMGMMHMMDBNKNNLDACARTPosNeuNegSupervisedLearning4thNovember2016Challenges Remain1.Text-Based:CapturethecompositionaleffectswithhigheraccuracyNegatingPositivesentencesNegatingNegativesentencesConjunction:2.Speech-Based:Effectivefeaturesunknown.EmotionalspeechsegmentstendtobetranscribedwithlowerASRaccuracy4thNovember2016Overview1.Introduction2.EmotionRecognitioninTextWordEmbeddingforSentimentAnalysisCNNforSentimentClassificationRNN,LSTMforsentimentClassificationPriorKnowledge+CNN/LSTMParsing+RNN3.EmotionRecognitioninSpeech4.EmotionRecognitioninConversations5.IndustrialApplicationHowdeeplearningcanchangethegame?4thNovember2016RANLP2015,HISSAR,BULGARIA6thSeptember201554Emotion Classification with Deep learning approaches4thNovember2016RANLP2015,HISSAR,BULGARIA1.Word Embedding as Features6thSeptember201555Representationoftextisveryimportantforperformanceofmanyreal-worldapplicationsincludingemotionrecognition:Localrepresentations:N-gramsBag-of-words1-of-NcodingContinuousRepresentations:LatentSemanticAnalysisLatentDirichletAllocationDistributedRepresentations:wordembeddingTomasMikolov,“LearningRepresentationsofTextusingNeuralNetworks”,NIPsDeeplearningWorkshop2013(Bengioetal.,2006;Collobert&Weston,2008;Mnih&Hinton,2008;Turianetal.,2010;Mikolovetal.,2013a;c)4thNovember2016RANLP2015,HISSAR,BULGARIA1.Word Embedding as Features6thSeptember201556Representationoftextisveryimportantforperformanceofmanyreal-worldapplicationsincludingemotionrecognition:Localrepresentations:N-gramsBag-of-words1-of-NcodingContinuousRepresentations:LatentSemanticAnalysisLatentDirichletAllocationDistributedRepresentations:wordembeddingTomasMikolov,“LearningRepresentationsofTextusingNeuralNetworks”,NIPsDeeplearningWorkshop20134thNovember2016RANLP2015,HISSAR,BULGARIAWord Embedding6thSeptember201557Skip-gramArchCBOWThehiddenlayervectoristheword-embeddingvectorforw(t)4thNovember2016Word Embedding for Sentiment DetectionIthasbeenwidelyacceptedasstandardfeaturesforNLPapplicationsincludingsentimentanalysissince2013Mikolov2013Thewordvectorspaceimplicitlyencodesmanylinguisticregularitiesamongwords:semanticandsyntacticExample:GooglePre-trainedwordvectorswith1000BillionwordsDoesitencodepolaritysimilarities?great0.729151bad0.719005terrific0.688912decent0.683735nice0.683609excellent0.644293fantastic0.640778better0.612073solid0.580604lousy0.576420 wonderful0.572612terrible0.560204Good0.558616Top Relevant Words to“good”MostlyYes,butitdoesntseparateantonymswell4thNovember2016RANLP2015,HISSAR,BULGARIALearning Sentiment-Specific Word Embedding6thSeptember201559Tang,etal,“LearningSentimentSpecificWordEmbeddingforTwitterSentimentClassification”,ACL20144thNovember2016RANLP2015,HISSAR,BULGARIALearning Sentiment-Specific Word Embedding6thSeptember201560Tang,etal,“LearningSentimentSpecificWordEmbeddingforTwitterSentimentClassification”,ACL2014InSpirit,itissimilartomulti-tasklearning.Itlearnsthesamewayastheregularword-embeddingwithlossfunctionconsideringbothsemanticcontextandsentimentdistancetothetwitteremotionsymbols.4thNovember201610milliontweetsselectedbypositiveandnegativeemoticonsastrainingdataTheTwittersentimentclassificationtrackofSemEval2013Learning Sentiment-Specific Word EmbeddingTang,etal,“LearningSentimentSpecificWordEmbeddingforTwitterSentimentClassification”,ACL20144thNovember2016Paragraph VectorsLeandMikolov,“DistributionalRepresentationsofSentencesandDocuments,ICML2014Paragraphvectorsaredistributionalvectorrepresentationforpiecesoftext,suchassentencesorparagraphsTheparagraphvectorsarealsoaskedtocontributetothepredictiontaskofthenextwordgivenmanycontextssampledfromtheparagraph.EachparagraphcorrespondstoonecolumninDItactsasamemoryrememberingwhatismissingfromthecurrentcontext,aboutthetopicoftheparagraph4thNovember2016Paragraph Vectors Best Results on MR Data SetLeandMikolov,“DistributionalRepresentationsofSentencesandDocuments,ICML20144thNovember2016Overview1.Introduction2.EmotionRecognitioninTextWordEmbeddingforSentimentAnalysisCNNforSentimentClassificationRNN,LSTMforsentimentClassificationPriorKnowledge+CNN/LSTMDatasetCollection3.EmotionRecognitioninSpeech4.EmotionRecognitioninConversations5.IndustrialApplication4thNovember2016 CNN for Sentiment Classification Ref:YoonKim.ConvolutionalNeuralNetworksforSentenceClassification.EMNLP,2014.4thNovember2016CNN for Sentiment Classification 1.Ref:YoonKim.ConvolutionalNeuralNetworksforSentenceClassification.EMNLP,2014.1.AsimpleCNNwithOneLayerofconvolutionontopofwordvectors.MotivatedbyCNNhasbeensuccessfulonmanyotherNLPtasks2.InputLayer:Wordvectorsarefrompre-trainedGoogle-Newsword2vector3.ConvLayer:Windowsize:3words,4words,5words.Eachwith100featuremap.300featuresinthepenultimatelayer4.PoolingLayer:MaxOvertimePoolingatthe5.Outputlayer:Fullyconnectedsoftmaxlayer,outputdistributionoverlabels6.Regularization:Drop-outonthepenultimatelayerwithaconstrainonthel2normsoftheweightvectors7.Fine-trainembeddingvectorsduringtraining4thNovember2016Common Datasets 4thNovember2016CNN for Sentiment Classification-Results CNN-rand:Randomlyinitializeallwordembeddings CNN-static:word2vec,keeptheembeddingsfixed CNN-nonstatic:Fine-tuningembeddingvectors4thNovember2016CNN for Sentiment Classification-Results 4thNovember2016Why it is successful?1.MultiplefiltersandmultiplefeaturemapsEmotionsareexpressedinsegments,insteadofthespanningoverthewholesentence2.Usepre-trainedword2vecvectorsasinputfeatures.3.Embeddingwordvectorsarefurtherimprovedfornon-statictraining.Antonymsarefurtherseparatedaftertraining.4thNovember2016Resources for This work 1.SourceCode:https:/ CNN for Sentiment Kalchbrenneretal,“AConvolutionalNeuralNetworkforModelingSentences”,ACL2014Hyper Parameters in Experiments:K=4m=5,14featuremapsm=7,6featuremapsd=484thNovember2016Dynamic CNN for Sentiment Kalchbrenneretal,“AConvolutionalNeuralNetworkforModelingSentences”,ACL2014One Dimension Convolution Two Dimension Convolution48 D word vectors randomly initiated 300 D Initiated with Google word2vectorMore complicated model architecture with dynamic poolingStraight Forward 6,4 feature maps100-128 feature maps4thNovember2016JohnsonandZhang.,“EffectiveUseofWordOrderforTextCategorizationwithConvolutionalNeuralNetworks”,ACL-2015Why CNN is effective Asimpleremedyistousewordbi-gramsinadditiontounigramsIthasbeennotedthatlossofwordordercausedbybag-of-wordvectors(bowvectors)isparticularlyproblematiconsentimentclassificationComparingSVMwithTri-gramfeatureswith1,2,3windowfilterCNNTop 100 FeaturesSVMCNNUni-Grams687Bi-Grams2833Tri-Grams460SVMs cant fully take advantage of high-order ngrams4thNovember2016Sentiment Classification Considering Features beyond Text with CNN ModelsTangetal.,“LearningSemanticRepresentationsofUsersandProductsforDocumentLevelSentimentClassification“”,ACL-20154thNovember2016Overview1.Introduction2.EmotionRecognitioninTextWordEmbeddingforSentimentAnalysisCNNforSentimentClassificationRNN,LSTMforsentimentClassificationPriorKnowledge+CNN/LSTMDatasetCollection3.EmotionRecognitioninSpeech4.EmotionRecognitioninConversations5.IndustrialApplication4thNovember2016 Recursive Neural Tensor NetworkSocheretal.,“RecursiveDeepModelsforSemanticCompositionalityoveraSentimentTreebank”,EMNLP-2013.http:/nlp.stanford.edu/sentiment/1.TheStanfordSentimentTreebackisacorpuswithfullylabeledparsetrees2.Createdtofacilitateanalysisofthecompositionaleffectsofsentimentinlanguage3.10,662sentencesfrommoviereviews.Parsedbystanfordparser.215,154phrasesarelabeled4.AmodelcalledRecursiveNeuralTensorNetworkswasproposed4thNovember2016 Recursive Neural Tensor Network-Distribution of sentiment values for N-gramsSocheretal.,“RecursiveDeepModelsforSemanticCompositionalityoveraSentimentTreebank”,EMNLP-2013.http:/nlp.stanford.edu/sentiment/Strongersentimentoftenbuildsupinlongerphrasesandthemajorityoftheshorterphrasesareneutral4thNovember2016 Recursive Neural Tensor Network(RNTN)Socheretal.,“RecursiveDeepModelsforSemanticCompositionalityoveraSentimentTreebank”,EMNLP-2013.http:/nlp.stanford.edu/sentiment/f=tanhVisthetensordirectlyrelateinputvectors,WistheregularRNNweightmatrix4thNovember2016Wa