大数据分析全英文 (28).pdf

《大数据分析全英文 (28).pdf》由会员分享，可在线阅读，更多相关《大数据分析全英文 (28).pdf（9页珍藏版）》请在淘文阁 - 分享文档赚钱的网站上搜索。

1、2Data Processing System Architecture Computing algorithmComputing ModelData processing systemComputing Platform&EngineComputing Platforms that provide various development kits and operating environmentsData storing systemData application systemComputing Models for different types of data,such as 1.B

2、atch Processing Model for massive data,MapReduce2.Stream Computing model for dynamic data streams,3.Large-scale concurrent processing(MPP)model for structured data4.large-scale physical memory In-memory Computing model;5.Data Flow Graph model;Computing Engine Hadoop,Spark,Storm,etcLarge-scale concur

3、rent processing(MPP-Massively Parallel Processing)model for Structured Data45Massively Parallel Processing 6Massively Parallel Processing 7Massively Parallel Processing Massively Parallel Processing is the coordinated processing of a single task by multiple processor,Each processor using its own OS

4、and memory and communicating with each other using some form of messaging interfaceMPP can be setup with a shared nothing or shared disk architectureData is horizontally partitioned across nodes such that each node has a sub set of rows form each table in the DBEach node then processes only rows on

5、its own disks.8MPP shared Nothing ArchitecturePerformance through segment instance parallelismMaster host and standby master HostMaster coordinates work with segment hostSegment host coordinates work with one or more segment instances.Segment instances process queries in parallelSegment hosts have t

6、heir own CPU disk memory(shared nothing)High speed interconnect for continuous pipelining of data processing9Massively Parallel Processing 10Greenplum1.Master Host Separate physical server with its own OS/CPU/storage/memory.2.Hosts master Database.no user data in master database but stores metadata

7、about segments think in terms of system tables.3.2,3,4 Segment hosts Individual physical servers with their own OS/CPU/storage/memory.4.Hosts segment Database.Each database stores portion of user data.5.Interconnect switch Segment server databases communicate through an interconnect switchThe main c

8、haracteristic of MPP database is data distribution.1.Data is distributed across each segment database to achieve data and processing parallelism.2.Creating a database table with DISTRIBUTED BY clause,data is automatically distributed across segment databases.In Greenplum you can either use hash or round-robin distribution.