hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From x...@apache.org
Subject [25/26] hadoop git commit: HDFS-12551. Ozone: Documentation: Add Ozone overview documentation. Contributed by Anu Engineer.
Date Mon, 02 Oct 2017 18:11:28 GMT
HDFS-12551. Ozone: Documentation: Add Ozone overview documentation. Contributed by Anu Engineer.

Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/2b6cb425
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/2b6cb425
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/2b6cb425

Branch: refs/heads/HDFS-7240
Commit: 2b6cb42582c40e55d6b9488f7caff58463ed5175
Parents: c8f9fea
Author: Xiaoyu Yao <xyao@apache.org>
Authored: Mon Oct 2 10:52:05 2017 -0700
Committer: Xiaoyu Yao <xyao@apache.org>
Committed: Mon Oct 2 11:00:37 2017 -0700

 .../src/site/markdown/OzoneOverview.md          | 88 ++++++++++++++++++++
 hadoop-project/src/site/site.xml                |  2 +
 2 files changed, 90 insertions(+)

diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/OzoneOverview.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/OzoneOverview.md
new file mode 100644
index 0000000..4dfd249
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/OzoneOverview.md
@@ -0,0 +1,88 @@
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+   http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+Ozone Overview
Ozone is an Object store for Apache Hadoop. It aims to scale to billions of
The following is a high-level overview of the core components of Ozone.

+![Ozone Architecture Overview](images/ozoneoverview.png) 

+The main elements of Ozone are
+### Clients
+Ozone ships with a set of ready-made clients. They are 
Ozone CLI and Corona.

+    * [Ozone CLI](./OzoneCommandShell.html) is the command line interface like 'hdfs' command.

+    * Corona is a  load generation tool for Ozone.

+### REST Handler
+Ozone provides both an RPC (Remote Procedure Call) as well as a  REST
+(Representational State Transfer) style interface. This allows clients to be
+written in many languages quickly. Ozone strives to maintain a similar
+interface between REST and RPC. The Rest handler offers the REST protocol
+services of Ozone.
+For most purposes, a client can make one line change to switch from REST to
+RPC or vice versa.  

+### Ozone File System
+Ozone file system (TODO: Add documentation) is a Hadoop compatible file system.
+This is the important user-visible component of ozone.
+This allows Hadoop services and applications like Hive/Spark to run against
+Ozone without any change.
+### Ozone Client
+This is like DFSClient in HDFS. This acts as the standard client to talk to
+Ozone. All other components that we have discussed so far rely on Ozone client
+(TODO: Add Ozone client documentation).

+### Key Space Manager

+Key Space Manager(KSM) takes care of the Ozone's namespace.
+All ozone entities like volumes, buckets and keys are managed by KSM
+(TODO: Add KSM documentation). In Short, KSM is the metadata manager for Ozone.
+KSM talks to blockManager(SCM) to get blocks and passes it on to the Ozone
+client.  Ozone client writes data to these blocks.
+KSM will eventually be replicated via Apache Ratis for High Availability.

+### Storage Container Manager
+Storage Container Manager (SCM) is the block and cluster manager for Ozone.
+SCM along with data nodes offer a service called 'containers'.
+A container is a group unrelated of blocks that are managed together
+as a single entity.
+SCM offers the following abstractions.

+![SCM Abstractions](images/scmservices.png)
+#### Blocks
+Blocks are like blocks in HDFS. They are replicated store of data.
+#### Containers
+A collection of blocks replicated and managed together.
+#### Pipelines
+SCM allows each container to choose its method of replication.
+For example, a container might decide that it needs only one copy of a  block
+and might choose a stand-alone pipeline. Another container might want to have
+a very high level of reliability and pick a RATIS based pipeline. In other
+words, SCM allows different kinds of replication strategies to co-exist.
+#### Pools
+A group of data nodes is called a pool. For scaling purposes,
+we define a pool as a set of machines. This makes management of datanodes
+#### Nodes
+The data node where data is stored.

diff --git a/hadoop-project/src/site/site.xml b/hadoop-project/src/site/site.xml
index 7716633..6e8cc04 100644
--- a/hadoop-project/src/site/site.xml
+++ b/hadoop-project/src/site/site.xml
@@ -107,6 +107,8 @@
     <menu name="Ozone" inherit="top">
       <item name="Getting Started" href="hadoop-project-dist/hadoop-hdfs/OzoneGettingStarted.html"/>
+      <item name="Ozone Overview"
+            href="hadoop-project-dist/hadoop-hdfs/OzoneOverview.html"/>
       <item name="Commands Reference" href="hadoop-project-dist/hadoop-hdfs/OzoneCommandShell.html"/>
       <item name="Ozone Rest API" href="hadoop-project-dist/hadoop-hdfs/OzoneRest.html"/>
       <item name="Ozone Metrics" href="hadoop-project-dist/hadoop-hdfs/OzoneMetrics.html"/>

To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org

View raw message