前向人工神经网络敏感性研究.ppt

《前向人工神经网络敏感性研究.ppt》由会员分享，可在线阅读，更多相关《前向人工神经网络敏感性研究.ppt（38页珍藏版）》请在淘文阁 - 分享文档赚钱的网站上搜索。

1、前向人工神经网络敏感性研究一一.引言引言 1.前向神经网络前向神经网络(FNN)FNN)介绍介绍神经元离散型：自适应线性元（Adaline）连续型：感知机（Perceptron）神经网络离散型：多层自适应线性网（Madaline）连续型：多层感知机（BP网或MLP）问题问题硬件精度对权的影响环境噪音对输入的影响动机动机参数的扰动对网络会产生怎样影响？如何衡量网络输出偏差的大小？2.研究提出研究提出建立网络输出与网络参数扰动之间的关系分析该关系，揭示网络的行为规律量化网络输出偏差3.研究内容研究内容指导网络设计，增强网络抗干扰能力度量网络性能，如容错和泛化能力研究其它网络课题的基础，如网络结构的

2、裁剪和参数的挑选等4.研究意义研究意义1.Madaline的敏感性n维几何模型（超球面）M.Stevenson,R.Winter,andB.Widrow,“SensitivityofFeedforwardNeuralNetworkstoWeightErrors,”IEEETrans.onNeural,Networks,vol.1,no.1,1990.统计模型（方差）S.W.Pich,“TheSelectionofWeightAccuraciesforMadalines,”IEEETrans.onNeuralNetworks,vol.6,no.2,1995.二二.研究纵览研究纵览（典型（典型方法

3、和方法和文献文献）分析方法（偏微分）S.Hashem,“SensitivityAnalysisforFeed-ForwardArtificialNeuralNetworkswithDifferentiableActivationFunctions”,Proc.ofIJCNN,vol.1,1992.统计方法（标准差）J.Y.Choi&C.H.Choi,“SensitivityAna-lysisofMultilayerPerceptronwithDiffer-entiableActivationFunctions,”IEEETrans.onNeuralNetworks,vol.3,no.1,199

4、2.2.MLP的敏感性输入属性筛选J.M.Zurada,A.Malinowski,S.Usui,“PerturbationMethodforDeletingRedundantInputsofPerceptronNetworks”,Neurocomputing,vol.14,1997.网络结构裁减A.P.Engelbrecht,“ANewPruningHeuristicBasedonVarianceAnalysisofSensitivityInformation”,IEEETrans.onNeuralNetworks,vol.12,no.6,2001.3.敏感性的应用J.L.Berniereta

5、l,“AQuantitiveStudyof FaultTolerance,NoiseImmunityandGeneralizationAbilityofMLPs,”Neural Computation,vol.12,2000.容错和泛化问题三三.研究方法研究方法1.自底向上自底向上方法方法单个神经元整个网络2.概率统计方法概率统计方法概率率（离散型）均值（连续型）3.n-维几何模型维几何模型超矩形的顶点（离散型）超矩形体（连续型）四四.已获成果（代表性论文）已获成果（代表性论文）敏感性分析：“SensitivityAnalysisofMultilayerPercep-trontoInputan

6、dWeightPerturbations,”IEEETrans.onNeuralNetworks,vol.12,no.6,pp.1358-1366,Nov.2001.敏感性量化：“AQuantifiedSensitivityMeasureforMulti-layerPerceptrontoInputPerturbation,”NeuralComputation,vol.15,no.1,pp.183-212,Jan.2003.隐层节点的裁剪（敏感性应用）：“HiddenNeuronPruningforMultilayerPerceptronsUsingSensitivityMeasure,”Pr

7、oc.ofIEEEICMLC2002,pp.1751-1757,Nov.2002.输入属性重要性的判定（敏感性应用）：“DeterminingtheRelevanceofInputFeaturesforMultilayerPerceptrons,”Proc.ofIEEESMC2003,Oct.2003.五五.未来工作未来工作进一步完善已有的结果进一步完善已有的结果,使之更加实用使之更加实用 放松限制条件 扩大分析范围 精确量化计算进一步应用所得的结果进一步应用所得的结果,解决实际问题解决实际问题探索新方法探索新方法,研究新类型的网络研究新类型的网络结束谢谢各位！谢谢各位！Effectsofin

8、put&weightdeviationsonneuronssensitivitySensitivity increases with input and weigh deviations,but the increase has an upper bound.EffectsofinputdimensiononneuronssensitivityThere exists an optimal value for the dimension of input,which yields the highest sensitivity value.Effectsofinput&weightdeviat

9、ionsonMLPssensitivitySensitivity of an MLP increases with the input and weight deviations.EffectsofthenumberofneuronsinalayerSensitivityofMLPs:n-2-2-1|1n10tothedimensionofinput.SensitivityofMLPs:2-n-2-1|1n10tothenumberofneuronsinthe1stlayer.SensitivityofMLPs:2-2-n-1|1n10tothenumberofneuronsinthe2ndl

10、ayer.There exists an optimal value for the number of neurons in a layer,which yields the highest sensitivity value.The nearer a layer to the output layer is,The more effect the number of neurons in the layer has.EffectsofthenumberoflayersSensitivityofMLPs:2-1,2-2-1,.,2-2-2-2-2-2-2-2-2-2-1tothenumber

11、oflayers.Sensitivity decreases with the number increasing,and the decrease almost levels off when the number becomes large.Sensitivityoftheneuronswith2-dimensionalinputSensitivityoftheneuronswith3-dimensionalinputSensitivityoftheneuronswith4-dimensionalinputSensitivityoftheneuronswith5-dimensionalin

12、putSensitivityoftheMLPs:2-2-1,2-3-1,2-2-2-1Simulation1(FunctionApproximation)ImplementanMLPtoapproximatethefunction:whereImplementationconsiderationsTheMLParchitectureisrestrictedto2-n-1.TheconvergenceconditionisMES-goal=0.01&Epoch105.Thelowesttrainablenumberofhiddenneuronsisn=5.Thepruningprocessess

13、tartwithMLPsof2-5-1andstopatanarchitectureof2-4-1.TherelevantdatausedbyandresultedfromthepruningprocessarelistedinTable1andTable2.TABLE1.Datafor3MLPswith5hiddenneuronstorealizethefunctionMLP2-5-1EpochMSE(training)MSE(testing)TrainedweightsandbiasMSE-(goal=0.01&epoch=100000)SensitivityRelevance130586

14、0.0009998160.0117005-12.9212-0.299933.7943-34.605731.4768-31.0169-0.5607-0.81401.1737-1.1026-5.450712.7341-13.0816-12.01718.7152bias=00.0317940.0022720.0014060.0270660.0018150.17330.02890.01840.32530.01582652090.0009999590.012457332.6223-33.3731-0.73610.7202-31.841231.2399-15.1872-0.0937-0.3989-1.00

15、2811.9959-15.490512.2103-6.0877-12.5057bias=00.0021760.0004630.0018210.0310170.0270680.02610.00720.02220.18880.33853260940.0009999440.0120354-15.094017.6184-19.916321.4109-14.0535-0.84601.0263-0.125826.7757-26.12598.8172-18.6532-6.830716.8506-10.4671bias=00.0135470.0066610.0262200.0283520.0023240.11

16、940.12420.17910.47770.0243TABLE2.Dataforthe3prunedMLPswith4hiddenneuronstorealizethefunctionMLP2-4-1EpochMSE(training)MSE(testing)Retrainedweightsandbias(goal=0.01&epoch=100000)SensitivityRelevance1(Obtainedbyremovingthe5thneuronfromtheMLPof2-5-1)22510.0009999980.0114834-14.4387-0.700334.8366-35.608

17、033.1285-32.6271-1.50650.0184-5.703613.0579-13.2457-12.1803bias=4.23490.0270140.0021000.0014600.0313430.15410.02740.01930.38182(Obtainedbyremovingthe2ndneuronfromtheMLPof2-5-1)19450.0009999210.011964533.5805-34.2727-32.931332.3172-15.8016-0.5610-1.33180.010312.626712.7961-6.1782-13.3652bias=-7.94680

18、.0019540.0018000.0269020.0292830.02470.02300.16620.39143(Obtainedbyremovingthe5thneuronfromtheMLPof2-5-1)132530.0009999710.011926-34.397433.8148-34.325034.7990-1.29090.019811.80970.887915.7984-15.6503-12.96066.0722bias=-1.41940.0016370.0013160.0288340.0281220.02590.02060.37370.1708Simulation2(Classi

19、fication)ImplementanMLPtosolvetheXORproblem:01ImplementationconsiderationsTheMLParchitectureisrestrictedto2-n-1.TheconvergenceconditionisMES-goal=0.1&Epoch105.ThepruningprocessesstartwithMLPsof2-5-1andstopatanarchitectureof2-4-1.TherelevantdatausedbyandresultedfromthepruningprocessarelistedinTable3a

20、ndTable4.TABLE3.Datafor3MLPswith5hiddenneuronstorealizethefunctionMLP2-5-1EpochMSE(training)MSE(testing)Trainedweightsandbias(goal=0.1&epoch=100000)SensitivityRelevance1445180.09999970.1092172.8188-8.11432.4420-0.54502.57663.70371.4955-2.9245-2.5714-3.712414.0153-43.990728.063619.5486-68.6432bias=00

21、.0475990.0357470.0315180.0273550.0315130.66711.57250.88450.53482.16322510980.09999980.1130061.4852-3.89021.06920.1466-1.0723-0.1455-7.03012.5695-3.1382-2.809423.9314-19.182427.156514.9694-91.6363bias=00.0375930.0201700.0201780.0455040.0325500.89970.38690.54800.68122.98283336310.09999940.113693.29202

22、.9094-1.00673.4724-7.05782.4377-3.2921-2.90961.5303-0.060645.7579-30.059816.5386-52.2874-29.7040bias=00.0314980.0391660.0462100.0314970.0317151.44131.17730.76421.64690.9421TABLE4.Dataforthe3prunedMLPswith4hiddenneuronstorealizethefunctionMLP2-4-1EpochMSE(training)MSE(testing)Retrainedweightsandbias(

23、goal=0.1&epoch=100000)SensitivityRelevance1(Obtainedbyremovingthe4thneuronfromtheMLPof2-5-1)226110.09999990.1090852.8745-6.88491.98440.04052.62953.8648-2.6270-3.865622.5649-51.345833.0982-74.4371bias=5.55700.0431730.0286270.0307080.0307170.97421.46991.01642.28652(Obtainedbyremovingthe2ndneuronfromth

24、eMLPof2-5-1)144570.09999980.1127921.1511-3.9352-1.4080-0.2348-6.82772.3307-3.2002-2.967026.366831.543716.3482-98.8089bias=-12.46560.0408410.0295910.0459790.0316121.07680.93340.75173.12353(Obtainedbyremovingthe3rdneuronfromtheMLPof2-5-1)175010.09999970.1114993.03863.7789-1.34714.6670-3.0386-3.77893.5143-0.757959.1526-34.0215-58.5949-36.1761bias=1.74740.0291140.0430970.0291140.0413721.72221.46621.70591.4967