2022年云存储-设计文件 .pdf

《2022年云存储-设计文件 .pdf》由会员分享，可在线阅读，更多相关《2022年云存储-设计文件 .pdf（20页珍藏版）》请在淘文阁 - 分享文档赚钱的网站上搜索。

1、 Design:云存储李建奇V0.11 2011-1-15 V0.20 2011-2-20 1 Executive view 1.1 云存储比专有存储的优势？扩展性在 PB 级别后，具有优势共享 pc 计算资源开放，可编程性，适应计算可以更靠近计算Low cost for better reliability&availability&performance 1.1.1Unstructured data 比如，图像但是也只有超过一定的规模后，才具有优势。这个规模临界点，我猜想目前可能在 2PB 左右。(400 台机器的cluster 往上)。每 GB 存储成本的下降，将继续降低这个规模临界点。

2、Haystack 28%cheaper than Nas,4X read throughtput than Nas.1.1.2Structured data 比如，email,webpage,user 等不过就存储计算，也就是structured data 来说，要对比的,倒不是 cost/GB.，存储计算所重视的是海量计算的可用性。1.2 与Cluster NAS 比较z好的 scalability zHigh availability 名师资料总结-精品资料欢迎下载-名师精心整理-第 1 页，共 20 页 -1.3 系统首次部署的开发代价应该小于12 人年可以考虑先部署一个 key/val

3、ue 存储应用。1.4 功能定义z这个存储框架，要为各种云存储提供支持?海量对象存储在离线应用，强调 bandwidth?海量对象存储在在线应用，强调 low latency.z设计的重点是关于分区，复制，transaction,failure recovery 等。在线的海量：?图像?Email,?文档数据库?No sql 数据库1.5 不支持的需求 (TBD)1.6 预防风险我们相信，尚且不知道的影响性能和稳定性的东西是存在的。z尽可能的充分设计z设计和实现要面向服务，组件，重构z实现要尽可能的采取简单策略z复杂系统的故障非常难以诊断，必须充分设计监控和调试部分1.7 产品线，未来zkey

4、/value zBlob,EBS zkey/value db,such as SimpleDB zColumn oriented db,such as Bigtable z。名师资料总结-精品资料欢迎下载-名师精心整理-第 2 页，共 20 页 -2Key feature 2.1 Key feature 2.1.1incremental 扩展2.1.2异构环境Capability aware of storage 2.1.3开放的框架，支持各种云存储应用2.1.4Tune the values of N,R and W to achieve their desired levels of pe

5、rformance,availability and durability 这个与 Cassandra 的 feature 是一样的。2.2 支持的应用2.2.1海量图片存储请求类似于，http:/.2.2.2海量文档数据库2.2.3海量structured data 2.2.4Blob 名师资料总结-精品资料欢迎下载-名师精心整理-第 3 页，共 20 页 -3 概念架构3.1 路线选择Amazon 的策略是，不同存储采用独立系统Google 的以 GFS(closorusu)为所有存储的基础选择 amazon 的结构。倾向于部署到虚拟机上。3.2 倾向于 S3 的 principle ht

6、tp:/ fully decentralized techniques to remove scaling bottlenecks and single points of failure.?Asynchrony:The system makes progress under all circumstances.?Autonomy:The system is designed such that individual components can make decisions based on local information.?Local responsibility:Each indiv

7、idual component is responsible for achieving its consistency;this is never the burden of its peers.?Controlled concurrency:Operations are designed such that no or limited concurrency control is required.?Failure tolerant:The system considers the failure of components to be a normal mode of operation

8、,and continues operation with no or minimal interruption.?Controlled parallelism:Abstractions used in the system are of such granularity that parallelism can be used to improve performance and robustness of recovery or the introduction of new nodes.?Decompose into small well-understood building bloc

9、ks:Do not try to provide a single service that does everything for every one,but instead build small components that can be used as building blocks for other services.?Symmetry:Nodes in the system are identical in terms of functionality,and require no or minimal node-specific configuration to functi

10、on.?Simplicity:The system should be made as simple as possible(-but no simpler).3.3 允许用户平衡 availability,consistency model 3.4 设计的思路名师资料总结-精品资料欢迎下载-名师精心整理-第 4 页，共 20 页 -自治的，无master 的思路倾向于运行于虚拟机上Dom 0EBS serviceNetworkipipiSCSI targetClientRoutingReplication图：写数据4 参考系统4.1 Dynamo 架构9Consistent hash 9 E

11、ventually consistent 4.2 HayStack Distributed object storage 名师资料总结-精品资料欢迎下载-名师精心整理-第 5 页，共 20 页 -A typical URL that directs the browser to the CDN looks like the following:http:/hCDNi/hCachei/hMachine idi/hLogical volume,Photoi haystack 是 low latency 的好设计。The key insight is to avoid disk operations

12、 when accessing metadata.Haystack provides a fault-tolerant and simple solution to photo storage at dramatically less cost and higher throughput than a traditional approach using NAS appliances.Furthermore,Haystack is incrementally scalable。zHaystack 存储是 log 结构。z用 XFS file system 名师资料总结-精品资料欢迎下载-名师精

13、心整理-第 6 页，共 20 页 -4.3 Amazon s3 4.4 Cassandra (TBD)4.5 Google gfs,Colossus,bigtable 4.5.1.1 GFS Constrains:Most modifications are appends Random writes are practically nonexistent Many files are written once,and read sequentially Two types of reads Large streaming reads Small random reads(in the for

14、ward direction)Sustained bandwidth more important than latency 名师资料总结-精品资料欢迎下载-名师精心整理-第 7 页，共 20 页 -因此，GFS 主要是针对离线部分设计的。The original Google File System,he says,didnt scale as well as the company would like.GFS lessons:zScaled to approximately 50M files,10P zLarge files increased upstream plexity zNo

15、t appropriate for latency sensitive applications zScaling limits added management overhead 4.5.1.2 Colossus9Next-generation cluster-level file system 9Automatically sharded metadata layer 9Data typically written using Reed-Solomon(1.5x)9Client-driven replication,encoding and replication 9Metadata sp

16、ace has enabled availability analyses 4.5.1.3 megastore 的 strong ACID 跨集群的存储，on bigtable 9Interactive online services Services like email,collaborative documents,and social networking have been growing exponentially and are testing the limits of existing infrastructure.9Using Paxos Megastore is the

17、largest sys-tem deployed that uses Paxos to replicate primary user data across datacenters on every write.So to improve availability and throughput we use multiple replicated logs,each governing its own partition of the 名师资料总结-精品资料欢迎下载-名师精心整理-第 8 页，共 20 页 -data set.9On bigtable 4.5.1.4 BigTable TBD

18、5 主要的设计决定5.1 Consistent hashing 类似 voldemort.node,partition.5.1.1Hash function Md5 5.1.2如何达到更好的负载均衡根据 dynamo 的 strategy 3.virtual node.5.1.3Partition 数量的权衡partition 的数量/node=3-？.主要是考虑 Routing table 不要太大merkle 的代价5.1.4Low latency 等距的 partition 好一些。可以1 hops routing.名师资料总结-精品资料欢迎下载-名师精心整理-第 9 页，共 20 页

19、-5.1.5颠簸如何扫描？有序的 key。等距的 partition，会导致，新的 partition 加入的时候，已有的节点变化太大。5.2 Replication&consistency 5.2.1Strong consistency:2PC commit 5.2.2Eventually Consistency 5.2.2.1 Versioning Vector clock，同 voldemort.5.2.2.2 冲突解决策略zLast write win zBy business logic 支持 Session consistency？支持 read-you-write？支持 mono

20、tonic read？5.2.2.3 Fault recovery 5.2.2.4 Temp failure&hinted hand off 同 voldemort 名师资料总结-精品资料欢迎下载-名师精心整理-第 10 页，共 20 页 -5.2.2.5 Permanent failure merkle,与 Cassandra 同。5.2.2.6 何时建立merkle tree?z批量建立。z需要时建立5.2.2.7 优化merkle tree?TODO 研究incremental merkle tree,5.3 混合语言，c+,java,python 工程简单。Java 对于大内存,回收影

21、响性能。Java 风险是 jvm 在 xen 上的性能和稳定性？9核心部分,c+9外围部分 java：比如监控9管理部分 python:9Web 部分：python or java 6 概要设计6.1 模块划分名师资料总结-精品资料欢迎下载-名师精心整理-第 11 页，共 20 页 -Serialization componentCommon componentRPCBlob EngineKey/value engineNo sql engineStorage serviceClient apiManagementMonitorEngine CommonSecurityCache servic

22、e6.2 Management:Membership 管理Hash 的基础是正确的membership,否则，写进去的东西，不知道到哪里读。Consistent hash 的结论是，可以在circle 前后找。6.2.1管理节点6.2.1.1 Node add/remove 的步骤zPush 新的 partition 信息z各个 Node 建立 hand-off 6.2.2 问题：membership 不一致，会不会导致 routing 错误？仅是暂时的。（必要的情况，可以中断服务，损失 availability）。名师资料总结-精品资料欢迎下载-名师精心整理-第 12 页，共 20 页 -6

23、.2.3问题：是否需要主动 Gossip-based membership 很少变动的cluster.不需要主动 gossip,既然 membership 可以不一致。6.3 Storage Service:routing设计6.3.1Routing table 6.3.1.1 一个physical node 负责多个 partition Cluster：Node Node Connection 6.3.1.2 数据结构partition 数组，2 hop routing,to node.Node 数组，维护 partitions.6.3.2Client connection Client 维

24、护与一个 physical node 的连接。6.4 storage service:hand-off,buffer,etc 设计?Simple Goal:Share storage to reduce costs?Typical scenario:?Pete runs video encoding using CPU&local disk?Roger runs a MapReduce that does heavy GFS reads 名师资料总结-精品资料欢迎下载-名师精心整理-第 13 页，共 20 页 -?Rafa runs seek-heavy Gmail on Bigtable w

25、/GFS?Andre runs seek-heavy Docs on Bigtable w/GFS?Things that go wrong:?Distribution of disks being accessed is not uniform?Non-storage system usage impacts CPU and disk?MapReduce impacts disks and buffer cache?GMail and Buzz both need hundreds of seeks NOW 6.4.1Buffer write and durable write 6.4.2B

26、ackground task with admission control 6.4.3让计算靠近存储的考虑返回的对象，带 Position 信息。6.4.4Failure detector 同 voldemort,原则上不专门detector,依赖 operation failure 判定。6.5 Storage engine 设计6.5.1Engine common:Share storage 6.5.2校验数据不能完全信赖 tcp 6.5.3Key/Value 类似 HayStack.在 XFS file system 上作。名师资料总结-精品资料欢迎下载-名师精心整理-第 14 页，共

27、20 页 -6.5.4Blob，EBS 设计 ISCSI 接口6.6 cache service设计http:/.6.7 Cache service 设计寻址模式，同 haystack.http:/6.7.1 仅仅是修饰 Cache 部分可以后设计。6.7.2解决cache spoiler to consistency(TBD)6.8 Security 设计Later 6.9 系统监控和诊断6.9.1暴露内部状态6.9.1.1 node 监控node 负载partition data size node data size 名师资料总结-精品资料欢迎下载-名师精心整理-第 15 页，共 20

28、页 -6.9.2监控采取 pull 策略，按需取用6.9.3可调试的关键的状态，可以在线更改6.10 Configuration service Node 起来后，从 service 获得 configuration.6.11 Client Api 6.11.1Client driven replication Pros:latency 被改善Cons:connection 数量被增加7 性能分析7.1 Scalability 分析针对的集群规模：2000 台 physical node 的集群。6T/node 12P 7.1.1Routing table 7.1.2Connection 每台

29、机器维护3000 个 connection？(TBD)名师资料总结-精品资料欢迎下载-名师精心整理-第 16 页，共 20 页 -7.1.3Merkle tree 计算密集？Partition 不要太大。网络密集？hint handoff，优先按需handoff,其次才是 background handoff.7.2 Latency 分析与 local storage 比较，latency 主要在 routing 环节，和replication 环节。7.2.1Routing Routing 的 latency,固定的,没有问题。7.2.2Cache Routing 待定7.2.3Replic

30、ation Write Replication 的 latency,采取 buffered writer。8 部署相关的考虑大规模集群的部署是很需要考虑的问题。8.1 Disk Raid 名师资料总结-精品资料欢迎下载-名师精心整理-第 17 页，共 20 页 -8.2 Network 机柜和交换机的安排。9 基础模块的设计Operating SystemExternal data representation ApplicationRequest reply protocol RMI,RPC and events Middlewarlayer9.1 RPC 9.1.1Api 隐藏掉协议，以便于替换名师资料总结-精品资料欢迎下载-名师精心整理-第 18 页，共 20 页 -9.1.2Reactor 9.2 Serialization Thrift,prococol-buffer 9.3 Common Component 9.3.1Compression 10Reference 名师资料总结-精品资料欢迎下载-名师精心整理-第 19 页，共 20 页 -名师资料总结-精品资料欢迎下载-名师精心整理-第 20 页，共 20 页 -