What Is Apache Hadoop

Posted 2020-08-24 rgqancy

What Is Apache Hadoop?

The Apache™ Hadoop® project develops open-source software for reliable, scalable, distributed computing.

Hadoop项目是为了开发可靠、可伸缩的分布式计算的开源软件。

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

Hadoop 软件库是一个框架，这个框架允许使用简单的编程模型就可以分布式地处理大量数据集，这些数据集本身是跨越多个集群的。Hadoop设计规模可从单机扩展到几千台机器，每台机器可提供本机的计算与存储能力。不像某些设计依赖硬件来达到高可用性，Hadoop 软件库本身在应用层被设计用来检测和处理故障。因此，它能提供一个在集群机器之上的高可用性服务，集群中的每台机器都可能遭遇故障。

 

The project includes these modules:

• Hadoop Common: The common utilities that support the other Hadoop modules.

　　通用的初始化组件，能够支持其他Hadoop模块。

• Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data.

　　Hadoop分布式文件系统：一套分布式文件系统，能够提供对应用数据的高穿透性访问能力。

• Hadoop YARN: A framework for job scheduling and cluster resource management.

　　YARN：一套任务调度和集群资源管理框架。

• Hadoop MapReduce: A YARN-based system for parallel processing of large data sets.

　　MapReduce：一套基于YARN的并行处理大量数据集的系统。

Other Hadoop-related projects at Apache include:

• Ambari™: A web-based tool for provisioning, managing, and monitoring Apache Hadoop clusters which includes support for Hadoop HDFS, Hadoop MapReduce, Hive, HCatalog, HBase, ZooKeeper, Oozie, Pig and Sqoop. Ambari also provides a dashboard for viewing cluster health such as heatmaps and ability to view MapReduce, Pig and Hive applications visually alongwith features to diagnose their performance characteristics in a user-friendly manner.
• Ambari，是一个基于web的配置，管理和监控阿帕奇的Hadoop集群的工具，它支持Hadoop HDFS，Hadoop MapReduce，Hive，HCatelog，HBase，ZooKeeper，Oozie，Pig和Soop。Ambari也提供一个仪表盘，这个仪表盘可以查看集群健康状况（例如热力图）以及以用户友好的方式，一站式可视化对MapReduce，Pig和Hive应用性能特性进行诊断的能力。
• Avro™: A data serialization system.
• Avro，一套序列化系统。
• Cassandra™: A scalable multi-master database with no single points of failure.
• Cassandra，一种可伸缩的多主机（主从机模式）数据库，可以避免单点故障。
• Chukwa™: A data collection system for managing large distributed systems.
• Chukwa,一套数据集合系统，可以管理大规模的分布式系统。
• HBase™: A scalable, distributed database that supports structured data storage for large tables.
• HBase，一套可伸缩的分布式数据库，可以支持大量数据表的结构化数据存储。
• Hive™: A data warehouse infrastructure that provides data summarization and ad hoc querying.
• Hive，一套数据仓库架构，可以提供数据摘要和约束即席查询。
• Mahout™: A Scalable machine learning and data mining library.
• Mahout，一套可伸缩的机器学习和数据挖掘库。
• Pig™: A high-level data-flow language and execution framework for parallel computation.
• Pig，一套高层数据流语言和执行框架，支持并行计算。
• Spark™: A fast and general compute engine for Hadoop data. Spark provides a simple and expressive programming model that supports a wide range of applications, including ETL, machine learning, stream processing, and graph computation.
• Spark，一套对Hadoop数据进行快速和通用计算的引擎。它提供一套简单并富有表现力的编程模型，支持一系列应用，包括ETL（Extract，Transform and Load），机器学习，流处理，以及图谱计算。
• Tez™: A generalized data-flow programming framework, built on Hadoop YARN, which provides a powerful and flexible engine to execute an arbitrary DAG of tasks to process data for both batch and interactive use-cases. Tez is being adopted by Hive™, Pig™ and other frameworks in the Hadoop ecosystem, and also by other commercial software (e.g. ETL tools), to replace Hadoop™ MapReduce as the underlying execution engine.
• Tez,一套泛型化的数据流编程框架，建立在Hadoop YARN之上。它能提供一个强大，弹性的引擎，这套引擎可以执行一个任意的DAG（有向无环图）任务，去处理批量的和可交互的用例。Tez已经被Hive，Pig和其他Hadoop生态系统的框架所采用，也被其他商业软件（例如ETL工具）用Hadoop MapReduce作为底层执行引擎使用。
• ZooKeeper™: A high-performance coordination service for distributed applications.
• ZooKeeper，是一项高性能的分布式应用的协同服务。

 引申：

HDFS，同类竞品有GFS（Google File System），亚马逊，阿里，腾讯，各自有自己命名的分布式文件系统。