云计算技术及应用ppt.ppt

《云计算技术及应用ppt.ppt》由会员分享，可在线阅读，更多相关《云计算技术及应用ppt.ppt（39页珍藏版）》请在淘文阁 - 分享文档赚钱的网站上搜索。

1、云计算技术及应用,大连理工大学计算机科学与技术学院2010年春季,基本情况,申彦明shenB810助教：齐恒clementqhB812Officehour:Fri3:30-4:30PMCoursewebsite:,教材内容,分布式系统的概况分布式与集群基本概念分布式数据库分布式文件系统GFS分布式编程MapReduce算法介绍搜索引擎与PageRank其它相关技术DataCenterBigTableAppEngine,Grading,HW：40%FinalProject:60%FinalprojectproposalProjectreports12teams,4-5students,Sylla

2、bus(Subjecttochange),Week2Mar8:Lecture1:IntroductionMar10:Lecture2:Map/ReduceTheoryandImplementation,HadoopWeek3Mar15:Lecture3&4:GuestSpeaker(8:00AM-11:35AM研教楼102)Mar17:Lecture5:DistributedFileSystemandtheGoogleFileSystemWeek4Mar22:Lecture6&7:GuestSpeaker(8:00AM-11:35AM研教楼102)Mar24:Lecture8:Distribu

3、tedGraphAlgorithmsandPageRankWeek5Mar29:Lecture9:IntroductiontoSomeProjectsMar31:Lecture10:DataCenters,Syllabus(Subjecttochange),Week6Apr5:Lecture11:SomeGoogleTechnologiesApr7:Lecture12:VirtualizationWeek7Lecture13&14:ProjectPresentationWeek8:NoclassWeek9:Lecture15&16:ProjectPresentation,GartnerRepo

4、rt,Top10StrategicTechnologyAreasfor2009VirtualizationCloudComputingServers:BeyondBladesWeb-OrientedArchitecturesEnterpriseMashupsSpecializedSystemsSocialSoftwareandSocialNetworkingUnifiedCommunicationsBusinessIntelligenceGreenInformationTechnology,Top10StrategicTechnologyAreasfor2010CloudComputingAd

5、vancedAnalyticsClientComputingITforGreenReshapingtheDataCenterSocialComputingSecurityActivityMonitoringFlashMemoryVirtualizationforAvailabilityMobileApplications,FromDesktop/HPC/GridstoInternetCloudsin30Years,HPCmovingfromcentralizedsupercomputerstogeographicallydistributeddesktops,clusters,andgrids

6、tocloudsoverlast30yearsR/DeffortsonHPC,clusters,Grids,P2P,andvirtualmachineshaslaidthefoundationofcloudcomputingthathasbeengreatlyadvocatedsince2007Locationofcomputinginfrastructureinareaswithlowercostsinhardware,software,datasets,space,andpowerrequirementsmovingfromdesktopcomputingtodatacenter-base

7、dclouds,WhatisCloudComputing?,1.Web-scaleproblems2.Largedatacenters3.Differentmodelsofcomputing4.Highly-interactiveWebapplications,1.“Web-Scale”Problems,Characteristics:Definitelydata-intensiveMayalsobeprocessingintensiveExamples:Crawling,indexing,searching,miningtheWebDatawarehousesSensornetworks“P

8、ost-genomics”lifesciencesresearchOtherscientificdata(physics,astronomy,etc.)Web2.0applications,Howmuchdata?,Googleprocesses20PBaday(2008)“allwordseverspokenbyhumanbeings”5EBCERNsLHCwillgenerate10-15PBayear,640Koughttobeenoughforanybody.,Whattodowithmoredata?,AnsweringfactoidquestionsPatternmatchingo

9、ntheWebWorksamazinglywellLearningrelationsStartwithseedinstancesSearchforpatternsontheWebUsingpatternstofindmoreinstances,HowdoImakemoney?,PetabytesofvaluablecustomerdataSittingidleinexistingdatawarehousesOverflowingoutofexistingdatawarehousesSimplybeingthrownawaySourceofdata:OLTPUserbehaviorlogsCal

10、l-centerlogsWebcrawls,publicdatasetsStructureddata(today)vs.unstructureddata(tomorrow)Howcananorganizationderivevaluefromallthisdata?,2.LargeDataCenters,Web-scaleproblems?Throwmoremachinesatit!CentralizationofresourcesinlargedatacentersNecessaryingredients:fiber,juice,andlandWhatdoOregon,Iceland,and

11、abandonedmineshaveincommon?ImportantIssues:EfficiencyRedundancyUtilizationSecurityManagementoverhead,3.DifferentComputingModels,UtilitycomputingWhybuymachineswhenyoucanrentcycles?Examples:AmazonsEC2PlatformasaService(PaaS)GivemeniceAPIandtakecareoftheimplementationExample:GoogleAppEngineSoftwareasaS

12、ervice(SaaS)Justrunitforme!Example:Gmail,“Whydoityourselfifyoucanpaysomeonetodoitforyou?”,4.WebApplications,Whatisthenatureoffuturesoftwareapplications?FromthedesktoptothebrowserSaaS=Web-basedapplicationsExamples:GoogleMaps,FacebookHowdowedeliverhighly-interactiveWeb-basedapplications?AJAX(asynchron

13、ousJavaScriptandXML)Ahackontopofamistakebuiltonsand,allheldtogetherbyducttapeandchewinggum?,SomeCloudDefinitions,IanFosteretaldefinedcloudcomputingasalarge-scaledistributedcomputingparadigm,thatisdrivenbyeconomicsofscale,inwhichapoolofabstractedvirtualized,dynamically-scalable,managedcomputingpower,

14、storage,platforms,andservicesaredeliveredondemandtoexternalcustomersovertheinternet(云计算是一种商业计算模型。它将计算任务分布在大量计算机构成的资源池上，使各种应用系统能够根据需要获取计算力、存储空间和各种软件服务。)IBMexpertsconsidercloudsthatcan:Hostavarietyofdifferentworkloads,includingbatch-stylebackendinteractive,user-facingapplicationsAllowworkloadstobedepl

15、oyedandscaled-outquicklythroughtherapidprovisioningofvirtualmachinesorphysicalmachinesSupportredundant,self-recovering,highlyscalableprogrammingmodelsthatallowworkloadstorecoverfromHW/SWfailuresMonitorresourceuseinrealtimetorebalanceallocationsondemand,InternetCloudGoals,Sharingofpeak-loadcapacityam

16、ongalargepoolofusers,improvingoverallresourceutilizationSeparationofinfrastructuremaintenancedutiesfromdomain-specificapplicationdevelopmentMajorcloudapplicationsincludeupgradedwebservices,distributeddatastorage,rawsupercomputing,andaccesstospecializedGrid,P2P,data-mining,andcontentnetworkingservice

17、s,ThreeAspectsinHardwarethatareNewinCloudComputing,Theillusionofinfinitecomputingresourcesavailableondemand,therebyeliminatingtheneedforclouduserstoplanfaraheadforprovisioningTheeliminationofanup-frontcommitmentbycloudusers,therebyallowingcompaniestostartsmallandincreasehardwareresourceswhenneededTh

18、eabilitytopaycomputingresourcesonashort-termbasisasneeded(e.g.,processorsbythehourandstoragebytheday)andreleasethemafterdoneandtherebyrewardingresourceconservation,SomeInnovativeCloudServicesandApplicationOpportunities,Smartandpervasivecloudapplicationsforindividuals,homes,communities,companies,andg

19、overnments,etc.CoordinatedCalendar,Itinerary,jobmanagement,events,andconsumerrecordmanagement(CRM)servicesCoordinatedwordprocessing,on-linepresentations,web-baseddesktops,sharingon-linedocuments,datasets,photos,video,anddatabases,etcDeployconventionalcluster,grid,P2P,socialnetworkingapplicationsincl

20、oudenvironments,morecost-effectivelyEarthboundApplicationsthatDemandElasticityandParallelismratherdatamovementCosts,OperationsinCloudComputing,UsersinteractwiththecloudtorequestserviceProvisioningtoolcarvesoutthesystemsfromthecloudconfigurationorreconfiguration,ordeprovisionTheserverscanbeeitherreal

21、orvirtualmachinesSupportingresourcesincludedistributedstoragesystem,datacenters,securitydevices,etc.,CloudComputingInstances,GoogleAmazonMicrosoftAzureIBMBlueCloud,GoogleCloudInfrastructure,Scheduler,Chubby,GFSmaster,Node,Node,Node,User,Application,Schedulerslave,GFSchunkserver,Linux,Node,MapReduceJ

22、ob,BigTableServer,GoogleCloudInfrastructure,AmazonElasticComputingCloud,SQS:SimpleQueueServiceEC2:RunningInstanceofVirtualMachinesEBS:ElasticBlockService,ProvidingtheBlockInterface,StoringVirtualMachineImagesS3:SimpleStorageService,SOAP,ObjectInterfaceSimpleDB:SimplifiedDatabase,MicrosoftAzurePlatfo

23、rm,Developer,Monitoring,ApplicationServer,ProvisioningManager,User,OpenSourceLinuxwithXen,TivoliMonitoringAgent,IBMBlueCloud,CostConsiderations:Power,Cooling,PhysicalPlant,andOperationalCosts,Costtechnologycostscostofsecurityetc.,Benefitsavailabilityopportunityconsolidationetc.,CostBreakdown,+Storag

24、e($/MByte/year)+Computing($/CPUCycles)+Networking($/bit),ResearchChallenges,ServiceavailabilityS3outage:authenticationserviceoverloadleadingtounavailabilityAppEnginepartialoutageprogrammingerrorGmail:siteunavailableSolutions:ThemanagementofaCloudComputingservicebyasinglecompanyresultsinasinglepointo

25、ffailure(SPF).IntheInternet,alargeISPusesmultiplenetworkproviderssothatfailurebyasinglecompanywillnottakethemofftheair.Similarly,weneedmultipleCloudComputingproviderstosupporteachothertoeliminateSPF.,ResearchChallenges,DataSecurityCurrentcloudofferingsareessentiallypublicratherthanprivatenetworks,ex

26、posingthesystemtomoreattackssuchasDDoSattacks.Solutions:Therearemanywellunderstoodtechnologiessuchasencryptedstorage,virtuallocalareanetworks,andnetworkmiddleboxes.,ResearchChallenges,DataTransferBottlenecksApplicationscontinuetobecomemoredata-intensive.Ifweassumeapplicationsmaybe“pulledapart”across

27、theboundariesofclouds,thismaycomplicatedataplacementandtransport.BothWANbandwidthandintra-cloudnetworkingtechnologyareperformancebottleneck.Industrialsolutions:Itisestimatedthat2/3ofthecostofWANbandwidthisconsumedbyhigh-endrouters,whereasonly1/3chargedbyfiberindustry.Wecanlowerthecostbyusingsimplerr

28、outersbuiltfromcommoditycomponentswithcentralizedcontrol,butresearchisheadingtowardsusinghigh-enddistributedrouters.,ResearchChallenges,SoftwareLicensingCurrentsoftwarelicensescommonlyrestrictthecomputersonwhichthesoftwarecanrun.Userspayforthesoftwareandthenpayanannualmaintenancefee.Manycloudcomputi

29、ngprovidersoriginallyreliedonopensourcesoftwareinpartbecausethelicensingmodelforcommercialsoftwareisnotagoodmatchtoUtilityComputing.Someideas:WecanencouragesalesforcesofsoftwarecompaniestosellproductsintoCloudComputing.Ortheycanimplementpay-per-usemodeltothesoftwaretoadapttoacloudenvironment.,Resear

30、chChallenges,ScalablestorageDifferencesbetweencommonstorageandcloudstorageThesystemisbuiltfrommanyinexpensivecommoditycomponentsthatoftenfailThesystemstoresamodestnumberoflargefilesTheworkloadsprimarilyconsistbothlargestreamingreadsandsmallrandomreads.Theworkloadsmanylarge,sequentialwritesthatappend

31、datatofilesandoncewritten,filesareseldommodifiedagain.Thecloudstorage(file)systemneedstosharemanyofthesamegoalsaspreviousdistributedfilesystemssuchasperformance,scalability,reliability,andavailability.Inaddition,itsdesignneedstobedrivenbykeyobservationsofthespecificworkloadsandtechnologicalenvironme

32、nt,bothcurrentandanticipated,thatreflectamarkeddeparturefromsomeearlierfilesystemdesignassumptions.GFSFilesaredividedintofixed-sizechunks,Chunksizeisoneofthekeydesignparameters.GFSchooses64MB,whichismuchlargerthantypicalfilesystemblocksizes.Themasterstoresthreemajortypesofmetadata:thefileandchunknam

33、espaces,themappingfromfilestochunks,andthelocationsofeachchunksreplicas.GFSsupportstheusualoperationstocreate,delete,open,close,read,andwritefiles.,ResearchChallenges,TransparentProgrammingModelProgramswrittenforcloudimplementationneedtobeautomaticallyparallelizedandexecutedonalargeclusterofcommodit

34、ymachines.Therun-timesystemshouldtakecareofthedetailsofpartitioningtheinputdata,schedulingtheprogramsexecutionacrossasetofmachines,handlingmachinefailures,andmanagingtherequiredinter-machinecommunication.Theprogrammingmodelshouldallowprogrammerswithoutmanyexperienceswithparallelanddistributedsystems

35、toeasilyutilizetheresourcesofalargedistributedsystem.MapReduceScalableDataProcessingonLargeClustersAwebprogrammingmodelimplementedforfastprocessingandgeneratinglargedatasetsAppliedmainlyinweb-scalesearchandcloudcomputingapplicationsUsersspecifyamapfunctiontogenerateasetofintermediatekey/valuepairsUs

36、ersuseareducefunctiontomergeallintermediatevalueswiththesameintermediatekey.,ResearchChallenges,SteveBallmersViewontheFutureofCloud,CloudcreatesopportunitiesandresponsibilitiesCloudlearnsandhelpsyoulearn,decideandtakeactionCloudenhancessocialandprofessionalinteractionsThecloudwantssmarterdevicesClou

37、ddrivesserveradvancesthat,inturn,drivethecloud,CloudComputingSkepticism,CLOUDCOMPUTING,Cloudcomputingissimplyabuzzwordusedtorepackagegridcomputingandutilitycomputing,bothofwhichhaveexistedfordecades.“Cloudcomputingissimplyabuzzwordusedtorepackagegridcomputingandutilitycomputing,bothofwhichhaveexiste

38、dfordecades.”,DefinitionofCloudComputing,LarryEllison,“Theinterestingthingaboutcloudcomputingisthatweveredefinedcloudcomputingtoincludeeverythingthatwealreadydo.Thecomputerindustryistheonlyindustrythatismorefashion-driventhanwomensfashion.MaybeImanidiot,butIhavenoideawhatanyoneistalkingabout.Whatisit?Itscompletegibberish.Itsinsane.Whenisthisidiocygoingtostop?”,