人工智能原理人工智能原理 (66).pdf

《人工智能原理人工智能原理 (66).pdf》由会员分享，可在线阅读，更多相关《人工智能原理人工智能原理 (66).pdf（30页珍藏版）》请在淘文阁 - 分享文档赚钱的网站上搜索。

1、Networked ModelsArtificial Intelligence5212.Models in Machine Learning 12.1.Probabilistic Models 12.2.Geometric Models 12.3.Logical Models 12.4.Networked ModelsContents:Artificial Intelligence:Learning:Models53 The networked models here refer to as the models of artificial neural network(ANN).这里的网络化

2、模型指的是人工神经网络模型(ANN)。An ANN is an artificial representation of the human brain that tries to simulate its learning processing.一个ANN是人脑的一种人工表征，试图模拟人类的学习过程。ANN can be constructed a system by interconnected“neurons”which send messages to each other.ANN可以通过互联的“神经元”构建一个系统，神经元之间相互发送消息。The connections betwee

3、n neurons have numeric weights that can be tuned based on experience,making ANN adaptive to inputs and capable of learning.神经元之间的连接具有数值权重，可以通过经验调整，使ANN适应输入并且能够学习。What are Networked Models 什么是网络化模型12.4.Networked ModelsArtificial Intelligence5412.4.Networked Models 12.4.1.Artificial Neural Networks 12

4、.4.2.Deep Neural NetworksContents:Artificial Intelligence:Learning:Models55Biological Neuron 生物神经元12.4.1.Artificial Neural NetworksInput 输入(Stimulus)刺激Output输出(Response)反应Dendrite树突Nucleus细胞核Axon轴突Synapse突触Artificial Intelligence:Learning:Models56Artificial Neuron 人工神经元12.4.1.Artificial Neural Netwo

5、rksNeuron i神经元Input 输入(Stimulus)刺激Output输出(Response)反应yiActivation Function激活函数Activation function:激活函数orx1x2x5x3x4wi1wi3wi2wi4wi5uiArtificial Intelligence:Learning:Models57Artificial Neural Network(ANN)人工神经网络12.4.1.Artificial Neural NetworksANN is a family of learning models inspired by biological

6、neural networksThe interconnection between the different layers of neuronsThe learning process for updating the weights of the interconnectionsThe activation function that converts a neurons weighted input to its outputInput 输入(Stimulus)刺激Output输出(Response)反应Hidden layers 隐藏层Input layers输入层Output la

7、yers输出层ANN是受生物神经网络启发的一系列学习模型不同的神经元层次之间互联学习过程是为了更新互联权重激活函数将神经元的加权输入转换为其输出Artificial Intelligence:Learning:Models581943,McCulloch and Pitts 马卡洛和匹茨 created a computational model for neural networks based on mathematics and algorithms called threshold logic.基于称之为阈值逻辑的数学和算法创建了神经网络的计算模型。1954,Farley and Cl

8、ark 法利和克拉克 first used computational machines,then called calculators,to simulate a Hebbian network.首次利用计算的机器、后来称其为计算器，来仿真赫布网络。1958,Rosenblatt 罗森布莱特 created perceptron,an algorithm for pattern recognition,which is with only one output layer,so also called“single layer perceptron”.创建了感知机，一种模式识别算法，它仅有一

9、个输出层，也被称为“单层感知机”。History of Artificial Neural Networks 人工神经网络的发展史12.4.1.Artificial Neural NetworksArtificial Intelligence:Learning:Models591969,Minsky and Papert明斯基和帕伯特 Published a famous book entitled“Perceptrons”.出版了一本名为“感知机”的著名书籍。It pointed in this book that the single layer perceptrons are only

10、capable of learning linearly separable patterns,but not possible to learn an XOR function.书中指出，单层感知机仅能学习线性可分模式，而不能用于学习异或功能。1974,Werbos韦伯斯 Proposed the back-propagationalgorithm,a method for training ANNs and used in conjunction with an optimization method such as gradient descent.提出了反向传播算法，一种用于训练ANN

11、s的方法，并且与梯度下降等优化方法结合使用。Regenerates interest in the 1980s.1980年代才引起重视。History of Artificial Neural Networks 人工神经网络的发展史12.4.1.Artificial Neural NetworksArtificial Intelligence:Learning:Models601989,Yann LeCunet al雅恩 勒昆等人 Published LeNet-5,a pioneering 7-level convolutional neural network(CNN)is applied

12、 to recognize hand-written numbers on checks.发表了LeNet-5，一种开拓性的7层卷积神经网络(CNN)，用于检查支票上的手写数字。1992,Schmidhuber施米德胡贝 Proposed recurrent neural network(RNN),this creates an internal state which allows it to exhibit dynamic temporal behavior.提出了循环神经网络，它创建网络的内部状态，得以展现动态时间行为。2006,Hinton and Salakhutdinov辛顿和萨拉

13、赫丁诺夫 Renewed interest in neural nets was sparked by the advent of deep learning.深度学习的出现，再次引发了对神经网络的兴趣。History of Artificial Neural Networks 人工神经网络的发展史12.4.1.Artificial Neural NetworksArtificial Intelligence:Learning:Models612012,Andrew Ng and Jeff Dean吴恩达和杰夫 迪恩 Google Brain team created a neural net

14、work that learned to recognize higher-level concepts,such as cats,from watching unlabeled images.Google大脑团队创建了一个神经网络，学会观看未标注图像来识别高层次概念，例如猫。2012,Krizhevskyet al 克利则夫斯基等 With Deep CNNs won the large-scale ImageNet competition by a significant margin over shallow machine learning methods.采用深度CNNs获得了大规模

15、ImageNet比赛的胜利，比浅层学习方法有显著优势。2014,Ian Goodfellow et al伊恩 古德菲勒等 Proposed generative adversarial network(GAN)which has two neural networks competing against each other in a zero-sum game framework.提出了生成对抗网络(GAN)，其中有两个神经网络，彼此以“零和”博弈方式相互竞争。History of Artificial Neural Networks 人工神经网络的发展史12.4.1.Artificial

16、Neural NetworksArtificial Intelligence:Learning:Models62Structures of Neural Networks 神经网络的结构12.4.1.Artificial Neural NetworksNeural Networks神经网络Feedforward Neural Networks前馈神经网络RecurrentNeural Networks循环神经网络Artificial Intelligence:Learning:Models63 Feedforward neural network 前馈神经网络 information move

17、s in only one direction,forward,from input nodes,through hidden nodes and to the output nodes.信息从输入结点仅仅以一个方向，即前进方向，穿过隐藏层并抵达输出节点。Recurrent neural network 循环神经网络 connections form a directed cycle.连接形成有向循环。creating an internal state of the network which allows it to exhibit dynamic temporal behavior.建立

18、网络的内部状态，使之展现动态的时间特性。Structures of Neural Network Models 神经网络模型的结构12.4.1.Artificial Neural NetworksFeedforward neural network 前馈神经网络Recurrent neural network 循环神经网络Artificial Intelligence:Learning:Models64 Back-propagation(BP)is an abbreviation for“Backward propagation of errors”.反向传播(BP)是“反向误差传播”的缩略语

19、。It is a common method of training Artificial Neural Networks,and used in conjunction with an optimization method such as gradient descent.是训练人工神经网络的常用方法，与梯度下降优化方法结合使用。The algorithm repeats a two phase cycle:该算法重复两个阶段的循环：Back-propagation 反向传播12.4.1.Artificial Neural Networksphase 1:propagation传播phas

20、e 2:weight update权值更新Repeat phase 1 and phase 2 until the performance of the network is satisfactory.重复阶段1和阶段2的操作，直到网络的性能得到满足。Artificial Intelligence:Learning:Models65 Phase 1:Propagation第1阶段：传播 Feedforward propagation前馈传播the input of training data through the neural network in order to generate out

21、put activations.输入的训练数据穿过神经网络，从而生成输出激活值。Back-propagation反向传播the output activations through the neural network using the training data target in order to generate the deltasof all output and hidden neurons.输出激活再使用训练数据目标穿过神经网络，生成所有的输出层和隐藏层神经元的差值。deltas=expected output-actual output values差值=期待输出-实际输出A

22、lgorithm of Back-propagation 反向传播算法12.4.1.Artificial Neural NetworksArtificial Intelligence:Learning:Models66 Phase 2:Weight update第2阶段：权值更新For each weight:对每个权值：Multiply its output delta and input activation,to get the gradient of the weight.将其输出差值与输入激活相乘，以便得到该权值梯度。Subtract a ratio(percentage)of th

23、e gradient from the weight.The ratio is called learning rate.从权值中减去梯度的比值（百分比）。该比值被称为学习率。The greater the ratio,the faster the neuron trains;比值越大，神经元训练越快；the lower the ratio,the more accurate the training is.比值越低，训练精度越高。Algorithm of Back-propagation 反向传播算法12.4.1.Artificial Neural NetworksArtificial In

24、telligence:Learning:Models67A Stochastic Gradient Descent Algorithm 随机梯度下降算法12.4.1.Artificial Neural NetworksfunctionSTOCHASTIC-GRADIENT-DESCENT()return the network initialize network weights(often small random values)dofor each training example named ex prediction=neural-net-output(network,ex)/forw

25、ard pass actual=teacher-output(ex)compute error(prediction-actual)at the output units compute whfor all weights from hidden layer to output layer/backward passcompute wifor all weights from input layer to hidden layerupdate network weights/input layer not modified by error estimate until all example

26、s classified correctly or another stopping criterion satisfied return the networkFor training a three-layer network(only one hidden layer)用于训练一个三层网络（仅有一个隐藏层）Artificial Intelligence:Learning:Models68(c)Good fit of the data数据良拟合Comparison of Training Results 训练结果的比较12.4.1.Artificial Neural Networks(a)

27、Under-fit of the data数据低拟合(b)Over-fit of the data数据过拟合Artificial Intelligence:Learning:Models69 There is no universally agreed upon threshold of depth dividing shallow neural networks from deep neural networks.就划分浅层神经网络与深层神经网络的深度而言，尚未有公认的观点。Shallow vs.Deep Neural Network 浅层与深层神经网络12.4.1.Artificial N

28、eural Networks But most researchers agree that deep neural networks have more than 2 of hidden layers,and hidden layers 10 to be very deep neural networks.但大多数研究人员认为，深度神经网络的隐藏层超过2、而隐藏层大于10的为超深度神经网络。Artificial Neural Network人工神经网络Input layer输入层Output layer输出层Hidden layers隐藏层Artificial Intelligence701

29、2.4.Networked Models 12.4.1.Artificial Neural Networks 12.4.2.Deep Neural NetworksContents:Artificial Intelligence:Learning:Models71 Biological:Visual cortex is Deep Hierarchical生物学：视觉皮层是深层次的Why Deep Hierarchy 为什么深度层次12.4.2.Deep Neural NetworksArtificial Intelligence:Learning:Models72 DNNs use many

30、layers of nonlinear processing units for feature extraction and transformation.DNNs使用许多层非线性处理单元，用于特征提取和转换。Able to learn multiple levels of features or representations of the data.Higher level features are derived from lower level features.能够学习数据的多层特征或表征。高层特征来自于低层特征。Be part of the broader machine lea

31、rning field:learning representations of data.成为更广泛的机器学习领域的一部分：学习数据表征。Learning multiple levels of representations that correspond to different levels of abstraction;the levels form a hierarchy of concepts.学习多层级表征，对应于不同的抽象层级；这种层级形成了一种概念的层次结构。Deep Neural Networks(DNNs)深度神经网络12.4.2.Deep Neural NetworksA

32、rtificial Intelligence:Learning:Models73Deep belief networks(DBN)Convolutional neural networks(CNN)Deep Boltzmann machines(DBM)Recurrent neural networks(RNN)Long short-term memory(LSTM)Auto-EncodersGenerative Adversarial Network(GAN)Typical Deep Neural Networks 代表性的深度神经网络12.4.2.Deep Neural Networks深

33、度信念网络(DBN)卷积神经网络(CNN)深度波兹曼机(DBM)循环网络(RNN)长短期记忆(LSTM)自动编码器生成对抗网络(GAN)Artificial Intelligence:Learning:Models74 CNN is a type of feed-forward artificial neural network that uses at least one of convolution in place of general matrix multiplication.CNN是一种前馈式人工神经网络，使用至少一个卷积层来代替一般的矩阵乘法。Case Study:Convolu

34、tional neural network(CNN)卷积神经网络12.4.2.Deep Neural Networks Four key ideas:四个关键思想：local connections(convolution)局部连接（卷积）shared weights共享权值 pooling(sampling)池化（采样）many layers.多层.Input输入Convolution卷积Pooling池化Fully conn.全连接Softmax分类器.Artificial Intelligence:Learning:Models75Convolution with a 3x3 filte

35、r用3x3滤波器进行卷积 Convolution layer卷积层Consist of a set of learnable filters,each filter is convolved across the width and height of the input volume,computing the dot product between the entries of the filter and the input,and producing a 2-dimensional activation map of that filter.包含一组学习滤波器，每个滤波器对输入的宽和高

36、进行卷积，计算滤波器和输入之间的点积，生成一个该滤波器的2维活动图。Pooling layer池化层A form of non-linear down-sampling.It partitions the input image into a set of non-overlapping rectangles and,for each such sub-region,outputs the maximum.是一种非线性下采样。它将输入图像分割成一组不重叠的矩形，对每个这样的子区域，再产出其最大值。Case Study:Convolutional neural network(CNN)卷积神经网

37、络12.4.2.Deep Neural NetworksMax pooling with a 2x2 filter and stride=2用2x2滤波器进行最大池化，步长=2Source:http:/ Intelligence:Learning:Models76Case Study:Generative Adversarial Network(GAN)生成对抗网络12.4.2.Deep Neural Networks GAN is pioneered by Ian Goodfellow et al at University of Montreal in 2014.GAN是由Goodfell

38、ow等人于2014年在蒙特利尔大学开创的。Adversarial Network,inspired by Adversarial game.对抗网络，灵感源于对抗博弈。Generator maps from a latent space to a particular data distribution of interest.生成器从潜在空间映射到所关注的特定数据分布。Discriminator discriminate between real samples and samples produced by generator.判别器在真实样本和生成样本之间进行判别。Training a

39、model in a worst-case scenario,with inputs chosen by an adversary.在最坏场景下训练模型，由adversary选择输入。Source:Goodfellow NIPS 2016 WorkshopArtificial Intelligence:Learning:Models77Case Study:Generative Adversarial Network(GAN)生成对抗网络12.4.2.Deep Neural NetworksThe generator worked well with digits(a)and faces(b)

40、,but it created very fuzzy and vague images(c)and(d)when using the CIFAR-10 dataset.该生成器对数字(a)和人脸(b)效果好，但使用CIFAR-10数据集时，却生成了非常模糊和含糊的图像(c)和(d)。Generative Adversarial Network(GAN)生成对抗网络Artificial Intelligence:Learning:Models78Speech recognitionObject recognitionImage retrievalImage understandingNatura

41、l language processingRecommendation systemsDrug discoveryBiomedical informaticsTypical Applications of Deep Neural Networks 深度神经网络的主要应用12.4.2.Deep Neural Networks语音识别物体识别图像检索图像理解自然语言处理推荐系统药物发现生物医学信息学Artificial Intelligence:Learning:Models79Deep Learning is a new area of Machine Learning research,which has been introduced with the objective of moving Machine Learning closer to one of its original goals:Artificial Intelligence.12.4.Networked ModelsSource:http:/ 后记