hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jh...@apache.org
Subject [07/29] hadoop git commit: HADOOP-14491. Azure has messed doc structure. Contributed by Mingliang Liu
Date Fri, 09 Jun 2017 03:04:22 GMT
HADOOP-14491. Azure has messed doc structure. Contributed by Mingliang Liu


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/536f0571
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/536f0571
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/536f0571

Branch: refs/heads/YARN-5734
Commit: 536f057158c445a57049f6c392869ae2f0be4f24
Parents: 6b5285b
Author: Mingliang Liu <liuml07@apache.org>
Authored: Mon Jun 5 18:24:40 2017 -0700
Committer: Mingliang Liu <liuml07@apache.org>
Committed: Tue Jun 6 11:09:28 2017 -0700

----------------------------------------------------------------------
 .../hadoop-azure/src/site/markdown/index.md     | 233 ++++++++++---------
 1 file changed, 123 insertions(+), 110 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/536f0571/hadoop-tools/hadoop-azure/src/site/markdown/index.md
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-azure/src/site/markdown/index.md b/hadoop-tools/hadoop-azure/src/site/markdown/index.md
index 1dca3b9..9c57e60 100644
--- a/hadoop-tools/hadoop-azure/src/site/markdown/index.md
+++ b/hadoop-tools/hadoop-azure/src/site/markdown/index.md
@@ -14,20 +14,9 @@
 
 # Hadoop Azure Support: Azure Blob Storage
 
-* [Introduction](#Introduction)
-* [Features](#Features)
-* [Limitations](#Limitations)
-* [Usage](#Usage)
-    * [Concepts](#Concepts)
-    * [Configuring Credentials](#Configuring_Credentials)
-    * [Page Blob Support and Configuration](#Page_Blob_Support_and_Configuration)
-    * [Atomic Folder Rename](#Atomic_Folder_Rename)
-    * [Accessing wasb URLs](#Accessing_wasb_URLs)
-    * [Append API Support and Configuration](#Append_API_Support_and_Configuration)
-    * [Multithread Support](#Multithread_Support)
-* [Testing the hadoop-azure Module](#Testing_the_hadoop-azure_Module)
-
-## <a name="Introduction" />Introduction
+<!-- MACRO{toc|fromDepth=1|toDepth=3} -->
+
+## Introduction
 
 The hadoop-azure module provides support for integration with
 [Azure Blob Storage](http://azure.microsoft.com/en-us/documentation/services/storage/).
@@ -38,7 +27,7 @@ on the additional artifacts it requires, notably the
 To make it part of Apache Hadoop's default classpath, simply make sure that
 HADOOP_OPTIONAL_TOOLS in hadoop-env.sh has 'hadoop-azure' in the list.
 
-## <a name="Features" />Features
+## Features
 
 * Read and write data stored in an Azure Blob Storage account.
 * Present a hierarchical file system view by implementing the standard Hadoop
@@ -54,15 +43,15 @@ HADOOP_OPTIONAL_TOOLS in hadoop-env.sh has 'hadoop-azure' in the list.
 * Tested on both Linux and Windows.
 * Tested at scale.
 
-## <a name="Limitations" />Limitations
+## Limitations
 
 * File owner and group are persisted, but the permissions model is not enforced.
   Authorization occurs at the level of the entire Azure Blob Storage account.
 * File last access time is not tracked.
 
-## <a name="Usage" />Usage
+## Usage
 
-### <a name="Concepts" />Concepts
+### Concepts
 
 The Azure Blob Storage data model presents 3 core concepts:
 
@@ -76,7 +65,7 @@ The Azure Blob Storage data model presents 3 core concepts:
   The internal implementation also uses blobs to persist the file system
   hierarchy and other metadata.
 
-### <a name="Configuring_Credentials" />Configuring Credentials
+### Configuring Credentials
 
 Usage of Azure Blob Storage requires configuration of credentials.  Typically
 this is set in core-site.xml.  The configuration property name is of the form
@@ -87,11 +76,12 @@ untrusted party.**
 
 For example:
 
-    <property>
-      <name>fs.azure.account.key.youraccount.blob.core.windows.net</name>
-      <value>YOUR ACCESS KEY</value>
-    </property>
-
+```xml
+<property>
+  <name>fs.azure.account.key.youraccount.blob.core.windows.net</name>
+  <value>YOUR ACCESS KEY</value>
+</property>
+```
 In many Hadoop clusters, the core-site.xml file is world-readable. It is possible to
 protect the access key within a credential provider as well. This provides an encrypted
 file format along with protection with file permissions.
@@ -110,14 +100,14 @@ For additional reading on the credential provider API see:
 
 ###### provision
 
-```
+```bash
 % hadoop credential create fs.azure.account.key.youraccount.blob.core.windows.net -value
123
     -provider localjceks://file/home/lmccay/wasb.jceks
 ```
 
 ###### configure core-site.xml or command line system property
 
-```
+```xml
 <property>
   <name>hadoop.security.credential.provider.path</name>
   <value>localjceks://file/home/lmccay/wasb.jceks</value>
@@ -127,7 +117,7 @@ For additional reading on the credential provider API see:
 
 ###### distcp
 
-```
+```bash
 % hadoop distcp
     [-D hadoop.security.credential.provider.path=localjceks://file/home/lmccay/wasb.jceks]
     hdfs://hostname:9001/user/lmccay/007020615 wasb://yourcontainer@youraccount.blob.core.windows.net/testDir/
@@ -145,22 +135,25 @@ specifies an external program to be invoked by Hadoop processes to decrypt
the
 key.  The encrypted key value is passed to this external program as a command
 line argument:
 
-    <property>
-      <name>fs.azure.account.keyprovider.youraccount</name>
-      <value>org.apache.hadoop.fs.azure.ShellDecryptionKeyProvider</value>
-    </property>
+```xml
+<property>
+  <name>fs.azure.account.keyprovider.youraccount</name>
+  <value>org.apache.hadoop.fs.azure.ShellDecryptionKeyProvider</value>
+</property>
 
-    <property>
-      <name>fs.azure.account.key.youraccount.blob.core.windows.net</name>
-      <value>YOUR ENCRYPTED ACCESS KEY</value>
-    </property>
+<property>
+  <name>fs.azure.account.key.youraccount.blob.core.windows.net</name>
+  <value>YOUR ENCRYPTED ACCESS KEY</value>
+</property>
 
-    <property>
-      <name>fs.azure.shellkeyprovider.script</name>
-      <value>PATH TO DECRYPTION PROGRAM</value>
-    </property>
+<property>
+  <name>fs.azure.shellkeyprovider.script</name>
+  <value>PATH TO DECRYPTION PROGRAM</value>
+</property>
+
+```
 
-### <a name="Page_Blob_Support_and_Configuration" />Page Blob Support and Configuration
+### Page Blob Support and Configuration
 
 The Azure Blob Storage interface for Hadoop supports two kinds of blobs,
 [block blobs and page blobs](http://msdn.microsoft.com/en-us/library/azure/ee691964.aspx).
@@ -182,10 +175,12 @@ folder names.
 
 For example:
 
-    <property>
-      <name>fs.azure.page.blob.dir</name>
-      <value>/hbase/WALs,/hbase/oldWALs,/data/mypageblobfiles</value>
-    </property>
+```xml
+<property>
+  <name>fs.azure.page.blob.dir</name>
+  <value>/hbase/WALs,/hbase/oldWALs,/data/mypageblobfiles</value>
+</property>
+```
 
 You can set this to simply / to make all files page blobs.
 
@@ -197,7 +192,7 @@ The configuration option `fs.azure.page.blob.extension.size` is the page
blob
 extension size.  This defines the amount to extend a page blob if it starts to
 get full.  It must be 128MB or greater, specified as an integer number of bytes.
 
-### <a name="Atomic_Folder_Rename" />Atomic Folder Rename
+### Atomic Folder Rename
 
 Azure storage stores files as a flat key/value store without formal support
 for folders.  The hadoop-azure file system layer simulates folders on top
@@ -216,12 +211,14 @@ the intention of the rename operation, to allow redo in event of a failure.
 
 For example:
 
-    <property>
-      <name>fs.azure.atomic.rename.dir</name>
-      <value>/hbase,/data</value>
-    </property>
+```xml
+<property>
+  <name>fs.azure.atomic.rename.dir</name>
+  <value>/hbase,/data</value>
+</property>
+```
 
-### <a name="Accessing_wasb_URLs" />Accessing wasb URLs
+### Accessing wasb URLs
 
 After credentials are configured in core-site.xml, any Hadoop component may
 reference files in that Azure Blob Storage account by using URLs of the following
@@ -238,28 +235,32 @@ For example, the following
 commands demonstrate access to a storage account named `youraccount` and a
 container named `yourcontainer`.
 
-    > hadoop fs -mkdir wasb://yourcontainer@youraccount.blob.core.windows.net/testDir
+```bash
+% hadoop fs -mkdir wasb://yourcontainer@youraccount.blob.core.windows.net/testDir
 
-    > hadoop fs -put testFile wasb://yourcontainer@youraccount.blob.core.windows.net/testDir/testFile
+% hadoop fs -put testFile wasb://yourcontainer@youraccount.blob.core.windows.net/testDir/testFile
 
-    > hadoop fs -cat wasbs://yourcontainer@youraccount.blob.core.windows.net/testDir/testFile
-    test file content
+% hadoop fs -cat wasbs://yourcontainer@youraccount.blob.core.windows.net/testDir/testFile
+test file content
+```
 
 It's also possible to configure `fs.defaultFS` to use a `wasb` or `wasbs` URL.
 This causes all bare paths, such as `/testDir/testFile` to resolve automatically
 to that file system.
 
-### <a name="Append_API_Support_and_Configuration" />Append API Support and Configuration
+### Append API Support and Configuration
 
 The Azure Blob Storage interface for Hadoop has optional support for Append API for
 single writer by setting the configuration `fs.azure.enable.append.support` to true.
 
 For Example:
 
-    <property>
-      <name>fs.azure.enable.append.support</name>
-      <value>true</value>
-    </property>
+```xml
+<property>
+  <name>fs.azure.enable.append.support</name>
+  <value>true</value>
+</property>
+```
 
 It must be noted Append support in Azure Blob Storage interface DIFFERS FROM HDFS SEMANTICS.
Append
 support does not enforce single writer internally but requires applications to guarantee
this semantic.
@@ -267,25 +268,29 @@ It becomes a responsibility of the application either to ensure single-threaded
 file path, or rely on some external locking mechanism of its own.  Failure to do so will
result in
 unexpected behavior.
 
-### <a name="Multithread_Support" />Multithread Support
+### Multithread Support
 
 Rename and Delete blob operations on directories with large number of files and sub directories
currently is very slow as these operations are done one blob at a time serially. These files
and sub folders can be deleted or renamed parallel. Following configurations can be used to
enable threads to do parallel processing
 
 To enable 10 threads for Delete operation. Set configuration value to 0 or 1 to disable threads.
The default behavior is threads disabled.
 
-    <property>
-      <name>fs.azure.delete.threads</name>
-      <value>10</value>
-    </property>
+```xml
+<property>
+  <name>fs.azure.delete.threads</name>
+  <value>10</value>
+</property>
+```
 
 To enable 20 threads for Rename operation. Set configuration value to 0 or 1 to disable threads.
The default behavior is threads disabled.
 
-    <property>
-      <name>fs.azure.rename.threads</name>
-      <value>20</value>
-    </property>
+```xml
+<property>
+  <name>fs.azure.rename.threads</name>
+  <value>20</value>
+</property>
+```
 
-### <a name="WASB_SECURE_MODE" />WASB Secure mode and configuration
+### WASB Secure mode and configuration
 
 WASB can operate in secure mode where the Storage access keys required to communicate with
Azure storage does not have to
 be in the same address space as the process using WASB. In this mode all interactions with
Azure storage is performed using
@@ -295,30 +300,32 @@ Romote mode, however for testing purposes the local mode can be enabled
to gener
 
 To enable Secure mode following property needs to be set to true.
 
-```
-    <property>
-      <name>fs.azure.secure.mode</name>
-      <value>true</value>
-    </property>
+```xml
+<property>
+  <name>fs.azure.secure.mode</name>
+  <value>true</value>
+</property>
 ```
 
 To enable SAS key generation locally following property needs to be set to true.
 
+```xml
+<property>
+  <name>fs.azure.local.sas.key.mode</name>
+  <value>true</value>
+</property>
 ```
-    <property>
-      <name>fs.azure.local.sas.key.mode</name>
-      <value>true</value>
-    </property>
-```
+
 To use the remote SAS key generation mode, an external REST service is expected to provided
required SAS keys.
 Following property can used to provide the end point to use for remote SAS Key generation:
 
+```xml
+<property>
+  <name>fs.azure.cred.service.url</name>
+  <value>{URL}</value>
+</property>
 ```
-    <property>
-      <name>fs.azure.cred.service.url</name>
-      <value>{URL}</value>
-    </property>
-```
+
 The remote service is expected to provide support for two REST calls ```{URL}/GET_CONTAINER_SAS```
and ```{URL}/GET_RELATIVE_BLOB_SAS```, for generating
 container and relative blob sas keys. An example requests
 
@@ -326,7 +333,8 @@ container and relative blob sas keys. An example requests
 ```{URL}/GET_CONTAINER_SAS?storage_account=<account_name>&container=<container>&relative_path=<relative
path>&sas_expiry=<expiry period>&delegation_token=<delegation token>```
 
 The service is expected to return a response in JSON format:
-```
+
+```json
 {
   "responseCode" : 0 or non-zero <int>,
   "responseMessage" : relavant message on failure <String>,
@@ -334,40 +342,42 @@ The service is expected to return a response in JSON format:
 }
 ```
 
-## <a name="WASB Authorization" />Authorization Support in WASB.
+### Authorization Support in WASB
 
 Authorization support can be enabled in WASB using the following configuration:
 
+```xml
+<property>
+  <name>fs.azure.authorization</name>
+  <value>true</value>
+</property>
 ```
-    <property>
-      <name>fs.azure.authorization</name>
-      <value>true</value>
-    </property>
-```
-  The current implementation of authorization relies on the presence of an external service
that can enforce
-  the authorization. The service is expected to be running on a URL provided by the following
config.
 
-```
-    <property>
-      <name>fs.azure.authorization.remote.service.url</name>
-      <value>{URL}</value>
-    </property>
+The current implementation of authorization relies on the presence of an external service
that can enforce
+the authorization. The service is expected to be running on a URL provided by the following
config.
+
+```xml
+<property>
+  <name>fs.azure.authorization.remote.service.url</name>
+  <value>{URL}</value>
+</property>
 ```
 
-  The remote service is expected to provide support for the following REST call: ```{URL}/CHECK_AUTHORIZATION```
-  An example request:
+The remote service is expected to provide support for the following REST call: ```{URL}/CHECK_AUTHORIZATION```
+An example request:
   ```{URL}/CHECK_AUTHORIZATION?wasb_absolute_path=<absolute_path>&operation_type=<operation
type>&delegation_token=<delegation token>```
 
-  The service is expected to return a response in JSON format:
-  ```
-  {
+The service is expected to return a response in JSON format:
+
+```json
+{
     "responseCode" : 0 or non-zero <int>,
     "responseMessage" : relevant message on failure <String>,
     "authorizationResult" : true/false <boolean>
-  }
-  ```
+}
+```
 
-## <a name="Testing_the_hadoop-azure_Module" />Testing the hadoop-azure Module
+## Testing the hadoop-azure Module
 
 The hadoop-azure module includes a full suite of unit tests.  Most of the tests
 will run without additional configuration by running `mvn test`.  This includes
@@ -382,10 +392,12 @@ that runs on a local machine.
 To use the emulator, install Azure SDK 2.3 and start the storage emulator.  Then,
 edit `src/test/resources/azure-test.xml` and add the following property:
 
-    <property>
-      <name>fs.azure.test.emulator</name>
-      <value>true</value>
-    </property>
+```xml
+<property>
+  <name>fs.azure.test.emulator</name>
+  <value>true</value>
+</property>
+```
 
 There is a known issue when running tests with the emulator.  You may see the
 following failure message:
@@ -399,6 +411,7 @@ file to `src/test/resources/azure-auth-keys.xml` and setting
 the name of the storage account and its access key.
 
 For example:
+
 ```xml
 <?xml version="1.0"?>
 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org


Mime
View raw message