大数据采集与清洗ppt课件.ppt

《大数据采集与清洗ppt课件.ppt》由会员分享，可在线阅读，更多相关《大数据采集与清洗ppt课件.ppt（36页珍藏版）》请在淘文阁 - 分享文档赚钱的网站上搜索。

1、为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能 数据采集与清洗数据采集与清洗 2019|02|15 周 乐为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能什么是大数据大数据处理流程大数据的主要特征大数据采集的概念大数据采集应用为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能1什么是大数据为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能为深入学习习近

2、平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能淘宝推荐 依据购物行为偏好依据购物行为偏好引荐引荐依据你最近的阅读依据你最近的阅读行为和消费行为进行为和消费行为进行引荐行引荐依据你用的设备往依据你用的设备往来不断猜特征来不断猜特征.依据时节改变进行依据时节改变进行引荐引荐为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,

3、贯彻全国教育大会精神,充分发挥中小学图书室育人功能2014-032015-082017-102016-032018大大数数据据工工作作首首先先写写入入政政府府工工作作报报告告十十三三五五规规划划纲纲要要提提出出实实施施国国家家大大数数据据战战略略 20182018年年政政府府工工作作报报告告提提出出：实实施施大大数数据据发发展展行行动动，注注重重用用互互联联网网、大大数数据据等等提升监管效能提升监管效能国国务务院院发发布布促促进进大大数数据据发发展展的的行动纲要行动纲要十十九九大大提提出出推推动动大大数数据据战战略略，与与实实体体经经济济深深度度融融合合行业现状与前景为深入学习习近平新时代中国

4、特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能2019年人社部拟最新发布15项新职业1.大数据工程技术人员大数据工程技术人员2.云计算工程技术人员云计算工程技术人员3.人工智能工程技术人员人工智能工程技术人员4.物联网工程技术人员物联网工程技术人员5.为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥

5、中小学图书室育人功能为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能什么是大数据大数据（大数据（Big DataBig Data）是指无法使用是指无法使用传统和常用的软件技术和工具在一定时传统和常用的软件技术和工具在一定时间内完成获取、管理和处理的数据集间内完成获取、管理和处理的数据集为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能2大数据的主要特征为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能大数据主要特征

6、 VolumeVolumeVelocityVelocityVarietyVarietyVeracityVeracity真实性（真实性（VeracityVeracity），），即追求高质量的数即追求高质量的数据。据。容量大（容量大（VolumeVolume）,指大规模的数据量，指大规模的数据量，并且数据量呈持续并且数据量呈持续增长趋势。增长趋势。速度快（速度快（VelocityVelocity）,指的是数据被创建指的是数据被创建和移动的速度。和移动的速度。种类多（种类多（VarietyVariety）,指数据来自多种数指数据来自多种数据源，数据种类和据源，数据种类和格式。格式。ValueValu

7、e价值密度低价值密度低（ValueValue），指随着），指随着数据量的增长，数数据量的增长，数据中有意义的信息据中有意义的信息却没有成相应比例却没有成相应比例增长。增长。为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能3大数据处理流程为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能大数据处理流程 数据预处理数据预处理 就是将采集就是将采集来的数据从多种数据库来的数据从多种数据库导入到大型的分布式数导入到大型的分布式数据库中（目前主要是据库中（目前主要是hfdshfds或

8、或hivehive）,并同时做并同时做一些简单的清洗和预处一些简单的清洗和预处理工作。理工作。数据统计分析数据统计分析 就是对上面就是对上面已经完成的存储在大型分已经完成的存储在大型分布式数据库中的数据进行布式数据库中的数据进行归类统计，可以满足一般归类统计，可以满足一般场景的分析需求。场景的分析需求。数据挖掘数据挖掘 是对数据进是对数据进行基于各种算法的分析行基于各种算法的分析计算，从而起到预测的计算，从而起到预测的效果，实现一些高级别效果，实现一些高级别数据分析的需求。数据分析的需求。数据采集数据采集 就是利用就是利用多种数据库（关系型，多种数据库（关系型，NOSQLNOSQL）去存储不）

9、去存储不同来源的数据。同来源的数据。数据展示数据展示 就是对就是对以上处理完的结果以上处理完的结果进行分析，或者形进行分析，或者形成报表。成报表。为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能4大数据采集的概念为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能大数据采集的概念3 3、大数据采集技术方法、大数据采集技术方法 大数据采集技术就是对数据进行大数据采集技术就是对数据进行 ETL ETL 操作，通过对数据进行提取、转换、加载，最操作，通过对数据进行提取、转换、加载，

10、最终挖掘数据的潜在价值。终挖掘数据的潜在价值。ETLETL指的是指的是Extract-Transform-LoadExtract-Transform-Load，也就是抽取、转换、加，也就是抽取、转换、加载。载。抽取抽取-从各种数据源获取数据从各种数据源获取数据 转换转换-按需求格式将源数据转换为目标数据按需求格式将源数据转换为目标数据 加载加载-把目标数据加载到数据仓库中把目标数据加载到数据仓库中2 2、数据采集与大数据采集的区别、数据采集与大数据采集的区别 传统数据采集：来源单一，数据量相当小；结构单一；关系数据库和并行数据库传统数据采集：来源单一，数据量相当小；结构单一；关系数据库和并行数

11、据库 大数据的数据采集：来源广泛，数量巨大；数据类型丰富；分布式数据库大数据的数据采集：来源广泛，数量巨大；数据类型丰富；分布式数据库1 1、什么是数据采集、什么是数据采集 数据采集就是数据获取，数据源主要分为线上数据和内容数据数据采集就是数据获取，数据源主要分为线上数据和内容数据为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能大数据采集系统1.日志采集系统（Apache Flume、Scribe）3.数据库采集系统（关系型、nosql等各种数据库）2.网络数据采集系统（Scrapy 框架、Apache Nutch）为深入学习习近平

12、新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能5大数据采集应用为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能技能准备PythonPython基础基础LinuxLinux操作系统基本操作操作系统基本操作数据库基础（数据库基础（SQLSQL语句操作）语句操作）为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能环境准备Py

13、thonPythonJdk(javaJdk(java环境环境)数据库（数据库（mysqlmysql）为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能Thanks 为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能YOUR TITLE Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult

14、to the man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to the man who to th

15、e man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.为深

16、入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能YOUR TITLE Nothing is difficult to the man who Nothing is difficult to the man who will try.Nothing is difficult to the will try.Nothing is difficult to the man who will try.Nothing is difficult man who will try.Nothing is difficult to the man who

17、 will try.Nothing is to the man who will try.Nothing is difficult to the man who will try.difficult to the man who will try.Nothing is difficult to the man who Nothing is difficult to the man who will try.Nothing is difficult to the will try.Nothing is difficult to the man who will try.Nothing is di

18、fficult man who will try.Nothing is difficult to the man who will try.Nothing is to the man who will try.Nothing is difficult to the man who will try.difficult to the man who will try.为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能2OKPPT工作室为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育

19、人功能YOUR TITLE Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothi

20、ng is difficult to the man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to t

21、he man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能YOUR TITLE Nothing is difficult to the Nothing is difficult to the man who will try.Nothing man who will try.No

22、thing is difficult to the man is difficult to the man who will try.who will try.Nothing is difficult to the Nothing is difficult to the man who will try.Nothing man who will try.Nothing is difficult to the man who is difficult to the man who will try.will try.Nothing is difficult to Nothing is diffi

23、cult to the man who will the man who will try.Nothing is difficult try.Nothing is difficult to the man who will to the man who will try.try.Nothing is difficult to the Nothing is difficult to the man who will try.Nothing man who will try.Nothing is difficult to the man who is difficult to the man wh

24、o will try.will try.Nothing is difficult to the Nothing is difficult to the man who will try.Nothing man who will try.Nothing is difficult to the man who is difficult to the man who will try.will try.为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能YOUR TITLE 21%9%28%42%为深入学习习近平新时代中国特色社会主义思想和党

25、的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能3OKPPT工作室为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能YOUR TITLE Nothing is difficult to the man who will try.Nothing is difficult to the man who will Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is diffic

26、ult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will tr

27、y.Nothing is difficult to the man who will Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the

28、man who will Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.try.为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能YOUR TITLE Nothing is difficult to the man who will try.Nothing is difficult to the man who will Nothing is difficult to the man who will

29、try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is Nothing is difficult to the difficult to the man who wil

30、l man who will try.try.Nothing is Nothing is difficult to the difficult to the man who will man who will try.try.Nothing is Nothing is difficult to the difficult to the man who will man who will try.try.Nothing is Nothing is difficult to the difficult to the man who will man who will try.try.为深入学习习近

31、平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能YOUR TITLE Nothing is difficult to the man who will Nothing is difficult to the man who will try.Nothing is difficult to the man who try.Nothing is difficult to the man who will try.Nothing is difficult to the man will try.Nothing is difficult to the m

32、an who will try.Nothing is difficult to the who will try.Nothing is difficult to the man who will try.Nothing is difficult to man who will try.Nothing is difficult to the man who will try.the man who will try.Nothing is difficult to the man who will Nothing is difficult to the man who will try.Nothi

33、ng is difficult to the man who try.Nothing is difficult to the man who will try.Nothing is difficult to the man will try.Nothing is difficult to the man who will try.Nothing is difficult to the who will try.Nothing is difficult to the man who will try.Nothing is difficult to man who will try.Nothing is difficult to the man who will try.the man who will try.A vs B为深入学习习近平新时代中国特色社会主义思想和党的十九大精神,贯彻全国教育大会精神,充分发挥中小学图书室育人功能Thanks