hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From a.@apache.org
Subject [07/11] hadoop git commit: HDFS-7668. Convert site documentation from apt to markdown (Masatake Iwasaki via aw)
Date Fri, 13 Feb 2015 02:20:24 GMT
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2f1e5dc6/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/TransparentEncryption.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/TransparentEncryption.apt.vm b/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/TransparentEncryption.apt.vm
deleted file mode 100644
index fe722fb..0000000
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/TransparentEncryption.apt.vm
+++ /dev/null
@@ -1,290 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~   http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
-  ---
-  Hadoop Distributed File System-${project.version} - Transparent Encryption in HDFS
-  ---
-  ---
-  ${maven.build.timestamp}
-
-Transparent Encryption in HDFS
-
-%{toc|section=1|fromDepth=2|toDepth=3}
-
-* {Overview}
-
-  HDFS implements <transparent>, <end-to-end> encryption.
-  Once configured, data read from and written to special HDFS directories is <transparently>
encrypted and decrypted without requiring changes to user application code.
-  This encryption is also <end-to-end>, which means the data can only be encrypted
and decrypted by the client.
-  HDFS never stores or has access to unencrypted data or unencrypted data encryption keys.
-  This satisfies two typical requirements for encryption: <at-rest encryption> (meaning
data on persistent media, such as a disk) as well as <in-transit encryption> (e.g. when
data is travelling over the network).
-
-* {Background}
-
-  Encryption can be done at different layers in a traditional data management software/hardware
stack.
-  Choosing to encrypt at a given layer comes with different advantages and disadvantages.
-
-    * <<Application-level encryption>>. This is the most secure and most flexible
approach. The application has ultimate control over what is encrypted and can precisely reflect
the requirements of the user. However, writing applications to do this is hard. This is also
not an option for customers of existing applications that do not support encryption.
-
-    * <<Database-level encryption>>. Similar to application-level encryption
in terms of its properties. Most database vendors offer some form of encryption. However,
there can be performance issues. One example is that indexes cannot be encrypted.
-
-    * <<Filesystem-level encryption>>. This option offers high performance, application
transparency, and is typically easy to deploy. However, it is unable to model some application-level
policies. For instance, multi-tenant applications might want to encrypt based on the end user.
A database might want different encryption settings for each column stored within a single
file.
-
-    * <<Disk-level encryption>>. Easy to deploy and high performance, but also
quite inflexible. Only really protects against physical theft.
-
-  HDFS-level encryption fits between database-level and filesystem-level encryption in this
stack. This has a lot of positive effects. HDFS encryption is able to provide good performance
and existing Hadoop applications are able to run transparently on encrypted data. HDFS also
has more context than traditional filesystems when it comes to making policy decisions.
-
-  HDFS-level encryption also prevents attacks at the filesystem-level and below (so-called
"OS-level attacks"). The operating system and disk only interact with encrypted bytes, since
the data is already encrypted by HDFS.
-
-* {Use Cases}
-
-  Data encryption is required by a number of different government, financial, and regulatory
entities.
-  For example, the health-care industry has HIPAA regulations, the card payment industry
has PCI DSS regulations, and the US government has FISMA regulations.
-  Having transparent encryption built into HDFS makes it easier for organizations to comply
with these regulations.
-
-  Encryption can also be performed at the application-level, but by integrating it into HDFS,
existing applications can operate on encrypted data without changes.
-  This integrated architecture implies stronger encrypted file semantics and better coordination
with other HDFS functions.
-
-* {Architecture}
-
-** {Overview}
-
-  For transparent encryption, we introduce a new abstraction to HDFS: the <encryption
zone>.
-  An encryption zone is a special directory whose contents will be transparently encrypted
upon write and transparently decrypted upon read.
-  Each encryption zone is associated with a single <encryption zone key> which is specified
when the zone is created.
-  Each file within an encryption zone has its own unique <data encryption key (DEK)>.
-  DEKs are never handled directly by HDFS.
-  Instead, HDFS only ever handles an <encrypted data encryption key (EDEK)>.
-  Clients decrypt an EDEK, and then use the subsequent DEK to read and write data.
-  HDFS datanodes simply see a stream of encrypted bytes.
-
-  A new cluster service is required to manage encryption keys: the Hadoop Key Management
Server (KMS).
-  In the context of HDFS encryption, the KMS performs three basic responsibilities:
-
-    [[1]] Providing access to stored encryption zone keys
-
-    [[1]] Generating new encrypted data encryption keys for storage on the NameNode
-
-    [[1]] Decrypting encrypted data encryption keys for use by HDFS clients
-
-  The KMS will be described in more detail below.
-
-** {Accessing data within an encryption zone}
-
-  When creating a new file in an encryption zone, the NameNode asks the KMS to generate a
new EDEK encrypted with the encryption zone's key.
-  The EDEK is then stored persistently as part of the file's metadata on the NameNode.
-
-  When reading a file within an encryption zone, the NameNode provides the client with the
file's EDEK and the encryption zone key version used to encrypt the EDEK.
-  The client then asks the KMS to decrypt the EDEK, which involves checking that the client
has permission to access the encryption zone key version.
-  Assuming that is successful, the client uses the DEK to decrypt the file's contents.
-
-  All of the above steps for the read and write path happen automatically through interactions
between the DFSClient, the NameNode, and the KMS.
-
-  Access to encrypted file data and metadata is controlled by normal HDFS filesystem permissions.
-  This means that if HDFS is compromised (for example, by gaining unauthorized access to
an HDFS superuser account), a malicious user only gains access to ciphertext and encrypted
keys.
-  However, since access to encryption zone keys is controlled by a separate set of permissions
on the KMS and key store, this does not pose a security threat.
-
-** {Key Management Server, KeyProvider, EDEKs}
-
-  The KMS is a proxy that interfaces with a backing key store on behalf of HDFS daemons and
clients.
-  Both the backing key store and the KMS implement the Hadoop KeyProvider API.
-  See the {{{../../hadoop-kms/index.html}KMS documentation}} for more information.
-
-  In the KeyProvider API, each encryption key has a unique <key name>.
-  Because keys can be rolled, a key can have multiple <key versions>, where each key
version has its own <key material> (the actual secret bytes used during encryption and
decryption).
-  An encryption key can be fetched by either its key name, returning the latest version of
the key, or by a specific key version.
-
-  The KMS implements additional functionality which enables creation and decryption of <encrypted
encryption keys (EEKs)>.
-  Creation and decryption of EEKs happens entirely on the KMS.
-  Importantly, the client requesting creation or decryption of an EEK never handles the EEK's
encryption key.
-  To create a new EEK, the KMS generates a new random key, encrypts it with the specified
key, and returns the EEK to the client.
-  To decrypt an EEK, the KMS checks that the user has access to the encryption key, uses
it to decrypt the EEK, and returns the decrypted encryption key.
-
-  In the context of HDFS encryption, EEKs are <encrypted data encryption keys (EDEKs)>,
where a <data encryption key (DEK)> is what is used to encrypt and decrypt file data.
-  Typically, the key store is configured to only allow end users access to the keys used
to encrypt DEKs.
-  This means that EDEKs can be safely stored and handled by HDFS, since the HDFS user will
not have access to unencrypted encryption keys.
-
-* {Configuration}
-
-  A necessary prerequisite is an instance of the KMS, as well as a backing key store for
the KMS.
-  See the {{{../../hadoop-kms/index.html}KMS documentation}} for more information.
-
-  Once a KMS has been set up and the NameNode and HDFS clients have been correctly configured,
an admin can use the <<<hadoop key>>> and <<<hdfs crypto>>>
command-line tools to create encryption keys and set up new encryption zones. Existing data
can be encrypted by copying it into the new encryption zones using tools like distcp.
-
-** Configuring the cluster KeyProvider
-
-*** dfs.encryption.key.provider.uri
-
-  The KeyProvider to use when interacting with encryption keys used when reading and writing
to an encryption zone.
-
-** Selecting an encryption algorithm and codec
-
-*** hadoop.security.crypto.codec.classes.EXAMPLECIPHERSUITE
-
-  The prefix for a given crypto codec, contains a comma-separated list of implementation
classes for a given crypto codec (eg EXAMPLECIPHERSUITE).
-  The first implementation will be used if available, others are fallbacks.
-
-*** hadoop.security.crypto.codec.classes.aes.ctr.nopadding
-
-  Default: <<<org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec,org.apache.hadoop.crypto.JceAesCtrCryptoCodec>>>
-
-  Comma-separated list of crypto codec implementations for AES/CTR/NoPadding.
-  The first implementation will be used if available, others are fallbacks.
-
-*** hadoop.security.crypto.cipher.suite
-
-  Default: <<<AES/CTR/NoPadding>>>
-
-  Cipher suite for crypto codec.
-
-*** hadoop.security.crypto.jce.provider
-
-  Default: None
-
-  The JCE provider name used in CryptoCodec.
-
-*** hadoop.security.crypto.buffer.size
-
-  Default: <<<8192>>>
-
-  The buffer size used by CryptoInputStream and CryptoOutputStream. 
-
-** Namenode configuration
-
-*** dfs.namenode.list.encryption.zones.num.responses
-
-  Default: <<<100>>>
-
-  When listing encryption zones, the maximum number of zones that will be returned in a batch.
-  Fetching the list incrementally in batches improves namenode performance.
-
-* {<<<crypto>>> command-line interface}
-
-** {createZone}
-
-  Usage: <<<[-createZone -keyName <keyName> -path <path>]>>>
-
-  Create a new encryption zone.
-
-*--+--+
-<path> | The path of the encryption zone to create. It must be an empty directory.
-*--+--+
-<keyName> | Name of the key to use for the encryption zone.
-*--+--+
-
-** {listZones}
-
-  Usage: <<<[-listZones]>>>
-
-  List all encryption zones. Requires superuser permissions.
-
-* {Example usage}
-
-  These instructions assume that you are running as the normal user or HDFS superuser as
is appropriate.
-  Use <<<sudo>>> as needed for your environment.
-
--------------------------
-# As the normal user, create a new encryption key
-hadoop key create myKey
-
-# As the super user, create a new empty directory and make it an encryption zone
-hadoop fs -mkdir /zone
-hdfs crypto -createZone -keyName myKey -path /zone
-
-# chown it to the normal user
-hadoop fs -chown myuser:myuser /zone
-
-# As the normal user, put a file in, read it out
-hadoop fs -put helloWorld /zone
-hadoop fs -cat /zone/helloWorld
--------------------------
-
-* {Distcp considerations}
-
-** {Running as the superuser}
-
-  One common usecase for distcp is to replicate data between clusters for backup and disaster
recovery purposes.
-  This is typically performed by the cluster administrator, who is an HDFS superuser.
-
-  To enable this same workflow when using HDFS encryption, we introduced a new virtual path
prefix, <<</.reserved/raw/>>>, that gives superusers direct access to the
underlying block data in the filesystem.
-  This allows superusers to distcp data without needing having access to encryption keys,
and also avoids the overhead of decrypting and re-encrypting data. It also means the source
and destination data will be byte-for-byte identical, which would not be true if the data
was being re-encrypted with a new EDEK.
-
-  When using <<</.reserved/raw>>> to distcp encrypted data, it's important
to preserve extended attributes with the {{-px}} flag.
-  This is because encrypted file attributes (such as the EDEK) are exposed through extended
attributes within <<</.reserved/raw>>>, and must be preserved to be able
to decrypt the file.
-  This means that if the distcp is initiated at or above the encryption zone root, it will
automatically create an encryption zone at the destination if it does not already exist.
-  However, it's still recommended that the admin first create identical encryption zones
on the destination cluster to avoid any potential mishaps.
-
-** {Copying between encrypted and unencrypted locations}
-
-  By default, distcp compares checksums provided by the filesystem to verify that the data
was successfully copied to the destination.
-  When copying between an unencrypted and encrypted location, the filesystem checksums will
not match since the underlying block data is different.
-  In this case, specify the {{-skipcrccheck}} and {{-update}} distcp flags to avoid verifying
checksums.
-
-* {Attack vectors}
-
-** {Hardware access exploits}
-
-  These exploits assume that attacker has gained physical access to hard drives from cluster
machines, i.e. datanodes and namenodes.
-
-  [[1]] Access to swap files of processes containing data encryption keys.
-
-        * By itself, this does not expose cleartext, as it also requires access to encrypted
block files.
-
-        * This can be mitigated by disabling swap, using encrypted swap, or using mlock to
prevent keys from being swapped out.
-
-  [[1]] Access to encrypted block files.
-
-        * By itself, this does not expose cleartext, as it also requires access to DEKs.
-
-** {Root access exploits}
-
-  These exploits assume that attacker has gained root shell access to cluster machines, i.e.
datanodes and namenodes.
-  Many of these exploits cannot be addressed in HDFS, since a malicious root user has access
to the in-memory state of processes holding encryption keys and cleartext.
-  For these exploits, the only mitigation technique is carefully restricting and monitoring
root shell access.
-
-  [[1]] Access to encrypted block files.
-
-        * By itself, this does not expose cleartext, as it also requires access to encryption
keys.
-
-  [[1]] Dump memory of client processes to obtain DEKs, delegation tokens, cleartext.
-
-        * No mitigation.
-
-  [[1]] Recording network traffic to sniff encryption keys and encrypted data in transit.
-
-        * By itself, insufficient to read cleartext without the EDEK encryption key.
-
-  [[1]] Dump memory of datanode process to obtain encrypted block data.
-
-        * By itself, insufficient to read cleartext without the DEK.
-
-  [[1]] Dump memory of namenode process to obtain encrypted data encryption keys.
-
-        * By itself, insufficient to read cleartext without the EDEK's encryption key and
encrypted block files.
-
-** {HDFS admin exploits}
-
-  These exploits assume that the attacker has compromised HDFS, but does not have root or
<<<hdfs>>> user shell access.
-
-  [[1]] Access to encrypted block files.
-
-        * By itself, insufficient to read cleartext without the EDEK and EDEK encryption
key.
-
-  [[1]] Access to encryption zone and encrypted file metadata (including encrypted data encryption
keys), via -fetchImage.
-
-        * By itself, insufficient to read cleartext without EDEK encryption keys.
-
-** {Rogue user exploits}
-
-  A rogue user can collect keys of files they have access to, and use them later to decrypt
the encrypted data of those files.
-  As the user had access to those files, they already had access to the file contents.
-  This can be mitigated through periodic key rolling policies.

http://git-wip-us.apache.org/repos/asf/hadoop/blob/2f1e5dc6/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/ViewFs.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/ViewFs.apt.vm b/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/ViewFs.apt.vm
deleted file mode 100644
index 4868a20..0000000
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/ViewFs.apt.vm
+++ /dev/null
@@ -1,304 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~   http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
-  ---
-  Hadoop Distributed File System-${project.version} - ViewFs Guide
-  ---
-  ---
-  ${maven.build.timestamp}
-
-ViewFs Guide
-
-%{toc|section=1|fromDepth=0}
-
-* {Introduction}
-
-  The View File System (ViewFs) provides a way to manage multiple Hadoop file system namespaces
(or namespace volumes).
-  It is particularly useful for clusters having multiple namenodes, and hence multiple namespaces,
in {{{./Federation.html}HDFS Federation}}.
-  ViewFs is analogous to <client side mount tables> in some Unix/Linux systems.
-  ViewFs can be used to create personalized namespace views and also per-cluster common views.
-
-  This guide is presented in the context of Hadoop systems that have several clusters, each
cluster may be federated into multiple namespaces.
-  It also describes how to use ViewFs in federated HDFS to provide a per-cluster global namespace
so that applications can operate in a way similar to the pre-federation world.
-
-* The Old World (Prior to Federation)
-
-** Single Namenode Clusters
-
-  In the old world prior to {{{./Federation.html}HDFS Federation}}, a cluster has a single
namenode which provides a single file system namespace for that cluster.
-  Suppose there are multiple clusters.
-  The file system namespaces of each cluster are completely independent and disjoint.
-  Furthermore, physical storage is NOT shared across clusters (i.e. the Datanodes are not
shared across clusters.)
-
-  The <<<core-site.xml>>> of each cluster has a configuration property
that sets the default file system to the namenode of that cluster:
-
-+-----------------
-  <property>
-    <name>fs.default.name</name>
-    <value>hdfs://namenodeOfClusterX:port</value>
-  </property>
-+-----------------
-
-  Such a configuration property allows one to use slash-relative names to resolve paths relative
to the cluster namenode.
-  For example, the path <<</foo/bar>>> is referring to <<<hdfs://namenodeOfClusterX:port/foo/bar>>>
using the above configuration.
-
-  This configuration property is set on each gateway on the clusters and also on key services
of that cluster such the JobTracker and Oozie.
-
-** Pathnames Usage Patterns
-
-  Hence on Cluster X where the <<<core-site.xml>>> is set as above, the
typical pathnames are
-
-  [[1]] <<</foo/bar>>>
-
-    * This is equivalent to <<<hdfs://namenodeOfClusterX:port/foo/bar>>>
as before.
-
-  [[2]] <<<hdfs://namenodeOfClusterX:port/foo/bar>>>
-
-    * While this is a valid pathname, one is better using <<</foo/bar>>>
as it allows the application and its data to be transparently moved to another cluster when
needed.
-
-  [[3]] <<<hdfs://namenodeOfClusterY:port/foo/bar>>>
-
-    * It is an URI for referring a pathname on another cluster such as Cluster Y.
-      In particular, the command for copying files from cluster Y to Cluster Z looks like:
-
-+-----------------
-distcp hdfs://namenodeClusterY:port/pathSrc hdfs://namenodeClusterZ:port/pathDest
-+-----------------
-
-  [[4]] <<<webhdfs://namenodeClusterX:http_port/foo/bar>>> and
-        <<<hftp://namenodeClusterX:http_port/foo/bar>>>
-
-    * These are file system URIs respectively for accessing files via the WebHDFS file system
and the HFTP file system.
-      Note that WebHDFS and HFTP use the HTTP port of the namenode but not the RPC port.
-
-  [[5]] <<<http://namenodeClusterX:http_port/webhdfs/v1/foo/bar>>> and
-        <<<http://proxyClusterX:http_port/foo/bar>>>
-
-    * These are HTTP URLs respectively for accessing files via {{{./WebHDFS.html}WebHDFS
REST API}} and HDFS proxy.
-
-** Pathname Usage Best Practices
-
-  When one is within a cluster, it is recommended to use the pathname of type (1) above instead
of a fully qualified URI like (2).
-  Fully qualified URIs are similar to addresses and do not allow the application to move
along with its data.
-
-* New World – Federation and ViewFs
-
-** How The Clusters Look
-
-  Suppose there are multiple clusters.
-  Each cluster has one or more namenodes.
-  Each namenode has its own namespace.
-  A namenode belongs to one and only one cluster.
-  The namenodes in the same cluster share the physical storage of that cluster.
-  The namespaces across clusters are independent as before.
-
-  Operations decide what is stored on each namenode within a cluster based on the storage
needs.
-  For example, they may put all the user data (<<</user/\<username\>>>>)
in one namenode, all the feed-data (<<</data>>>) in another namenode, all
the projects (<<</projects>>>) in yet another namenode, etc.
-
-** A Global Namespace Per Cluster Using ViewFs
-
-  In order to provide transparency with the old world, the ViewFs file system (i.e. client-side
mount table) is used to create each cluster an independent cluster namespace view, which is
similar to the namespace in the old world.
-  The client-side mount tables like the Unix mount tables and they mount the new namespace
volumes using the old naming convention.
-  The following figure shows a mount table mounting four namespace volumes <<</user>>>,
<<</data>>>, <<</projects>>>, and <<</tmp>>>:

-
-[./images/viewfs_TypicalMountTable.png]
-
-  ViewFs implements the Hadoop file system interface just like HDFS and the local file system.
-  It is a trivial file system in the sense that it only allows linking to other file systems.
-  Because ViewFs implements the Hadoop file system interface, it works transparently Hadoop
tools.
-  For example, all the shell commands work with ViewFs as with HDFS and local file system.
-
-  The mount points of a mount table are specified in the standard Hadoop configuration files.
-  In the configuration of each cluster, the default file system is set to the mount table
for that cluster as shown below (compare it with the configuration in {{Single Namenode Clusters}}).
-
-+-----------------
-  <property>
-    <name>fs.default.name</name>
-    <value>viewfs://clusterX</value>
-  </property>
-+-----------------
-
-  The authority following the <<<viewfs://>>> scheme in the URI is the
mount table name.
-  It is recommanded that the mount table of a cluster should be named by the cluster name.
-  Then Hadoop system will look for a mount table with the name "clusterX" in the Hadoop configuration
files.
-  Operations arrange all gateways and service machines to contain the mount tables for ALL
clusters such that, for each cluster, the default file system is set to the ViewFs mount table
for that cluster as described above.
-
-
-** Pathname Usage Patterns
-
-  Hence on Cluster X, where the <<<core-site.xml>>> is set to make the
default fs to use the mount table of that cluster, the typical pathnames are
-
-  [[1]] <<</foo/bar>>>
-
-    * This is equivalent to <<<viewfs://clusterX/foo/bar>>>.
-      If such pathname is used in the old non-federated world, then the transition to federation
world is transparent. 
-
-  [[2]] <<<viewfs://clusterX/foo/bar>>>
-
-    * While this a valid pathname, one is better using <<</foo/bar>>> as
it allows the application and its data to be transparently moved to another cluster when needed.
-
-  [[3]] <<<viewfs://clusterY/foo/bar>>>
-
-    * It is an URI for referring a pathname on another cluster such as Cluster Y.
-      In particular, the command for copying files from cluster Y to Cluster Z looks like:
-
-+-----------------
-distcp viewfs://clusterY:/pathSrc viewfs://clusterZ/pathDest
-+-----------------
-
-  [[4]] <<<viewfs://clusterX-webhdfs/foo/bar>>> and
-        <<<viewfs://clusterX-hftp/foo/bar>>>
-
-    * These are URIs respectively for accessing files via the WebHDFS file system and the
HFTP file system.
-
-  [[5]] <<<http://namenodeClusterX:http_port/webhdfs/v1/foo/bar>>> and
-        <<<http://proxyClusterX:http_port/foo/bar>>>
-
-    * These are HTTP URLs respectively for accessing files via {{{./WebHDFS.html}WebHDFS
REST API}} and HDFS proxy.
-      Note that they are the same as before.
-
-** Pathname Usage Best Practices
-
-  When one is within a cluster, it is recommended to use the pathname of type (1) above instead
of a fully qualified URI like (2).
-  Futher, applications should not use the knowledge of the mount points and use a path like
<<<hdfs://namenodeContainingUserDirs:port/joe/foo/bar>>> to refer to a file
in a particular namenode.
-  One should use  <<</user/joe/foo/bar>>> instead.
-
-** Renaming Pathnames Across Namespaces
-
-  Recall that one cannot rename files or directories across namenodes or clusters in the
old world.
-  The same is true in the new world but with an additional twist.
-  For example, in the old world one can perform the commend below.
- 
-+-----------------
-rename /user/joe/myStuff /data/foo/bar
-+-----------------
-
-  This will NOT work in the new world if <<</user>>> and <<</data>>>
are actually stored on different namenodes within a cluster.
-
-** FAQ
-
-  [[1]] <<As I move from non-federated world to the federated world, I will have to
keep track of namenodes for different volumes; how do I do that?>>
-
-  No, you won’t.
-  See the examples above – you are either using a relative name and taking advantage of
the default file system, or changing your path from <<<hdfs://namenodeCLusterX/foo/bar>>>
to <<<viewfs://clusterX/foo/bar>>>. 
-
-  [[2]] <<What happens of Operations move some files from one namenode to another namenode
within a cluster?>>
-
-  Operations may move files from one namenode to another in order to deal with storage capacity
issues.
-  They will do this in a way to avoid applications from breaking.
-  Let's take some examples.
-
-    * Example 1:
-    <<</user>>> and <<</data>>> were on one namenode
and later they need to be on separate namenodes to deal with capacity issues.
-    Indeed, operations would have created separate mount points for <<</user>>>
and <<</data>>>.
-    Prior to the change the mounts for <<</user>>> and <<</data>>>
would have pointed to the same namenode, say <<<namenodeContainingUserAndData>>>.
-    Operations will update the mount tables so that the mount points are changed to <<<namenodeContaingUser>>>
and <<<namenodeContainingData>>>, respectively.
-
-    * Example 2:
-    All projects were fitted on one namenode and but later they need two or more namenodes.
ViewFs allows mounts like <<</project/foo>>> and <<</project/bar>>>.
-    This allows mount tables to be updated to point to the corresponding namenode.
-
-  [[3]] <<Is the mount table in each>> <<<core-site.xml>>>
<<or in a separate file of its own?>>
-
-  The plan is to keep the mount tables in separate files and have the <<<core-site.xml>>>
{{{http://www.w3.org/2001/XInclude}xincluding}} it.
-  While one can keep these files on each machine locally, it is better to use HTTP to access
it from a central location.
-
-  [[4]] <<Should the configuration have the mount table definitions for only one cluster
or all clusters?>>
-
-  The configuration should have the mount definitions for all clusters since one needs to
have access to data in other clusters such as with distcp.
-
-  [[5]] <<When is the mount table actually read given that Operations may change a
mount table over time?>>
-
-  The mount table is read when the job is submitted to the cluster.
-  The <<<XInclude>>> in <<<core-site.xml>>> is expanded
at job submission time.
-  This means that if the mount table are changed then the jobs need to be resubmitted.
-  Due to this reason, we want to implement merge-mount which will greatly reduce the need
to change mount tables.
-  Further, we would like to read the mount tables via another mechanism that is initialized
at job start time in the future. 
-
-  [[6]] <<Will JobTracker (or Yarn’s Resource Manager) itself use the ViewFs?>>
-
-   No, it does not need to.
-   Neither does the NodeManager.
-
-  [[7]] <<Does ViewFs allow only mounts at the top level?>>
-
-  No; it is more general.
-  For example, one can mount <<</user/joe>>> and <<</user/jane>>>.
-  In this case, an internal read-only directory is created for <<</user>>>
in the mount table.
-  All operations on <<</user>>> are valid except that <<</user>>>
is read-only.
-
-  [[8]] <<An application works across the clusters and needs to persistently store
file paths.
-         Which paths should it store?>>
-
-  You should store <<<viewfs://cluster/path>>> type path names, the same
as it uses when running applications.
-  This insulates you from movement of data within namenodes inside a cluster as long as operations
do the moves in a transparent fashion.
-  It does not insulate you if data gets moved from one cluster to another; the older (pre-federation)
world did not protect you form such data movements across clusters anyway.
-
-  [[9]]  <<What about delegation tokens?>>
-
-  Delegation tokens for the cluster to which you are submitting the job (including all mounted
volumes for that cluster’s mount table), and for input and output paths to your map-reduce
job (including all volumes mounted via mount tables for the specified input and output paths)
are all handled automatically.
-  In addition, there is a way to add additional delegation tokens to the base cluster configuration
for special circumstances.
-
-* Appendix: A Mount Table Configuration Example
-
-  Generally, users do not have to define mount tables or the <<<core-site.xml>>>
to use the mount table.
-  This is done by operations and the correct configuration is set on the right gateway machines
as is done for <<<core-site.xml>>> today.
-
-  The mount tables can be described in <<<core-site.xml>>> but it is better
to use indirection in <<<core-site.xml>>> to reference a separate configuration
file, say <<<mountTable.xml>>>.
-  Add the following configuration element to <<<core-site.xml>>> for referencing
<<<mountTable.xml>>>:
-
-+-----------------
-<configuration xmlns:xi="http://www.w3.org/2001/XInclude"> 
-  <xi:include href="mountTable.xml" />
-</configuration> 
-+-----------------
-
-  In the file <<<mountTable.xml>>>, there is a definition of the mount
table "ClusterX" for the hypothetical cluster that is a federation of the three namespace
volumes managed by the three namenodes
-
-  [[1]] nn1-clusterx.example.com:8020,
-
-  [[2]] nn2-clusterx.example.com:8020, and
-
-  [[3]] nn3-clusterx.example.com:8020.
-
-  []
-
-  Here <<</home>>> and <<</tmp>>> are in the namespace
managed by namenode nn1-clusterx.example.com:8020,
-  and projects <<</foo>>> and <<</bar>>> are hosted on
the other namenodes of the federated cluster.  
-  The home directory base path is set to <<</home>>>
-  so that each user can access its home directory using the getHomeDirectory() method defined
in
-  {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}/{{{../../api/org/apache/hadoop/fs/FileContext.html}FileContext}}.
-
-+-----------------
-<configuration>
-  <property>
-    <name>fs.viewfs.mounttable.ClusterX.homedir</name>
-    <value>/home</value>
-  </property>
-  <property>
-    <name>fs.viewfs.mounttable.ClusterX.link./home</name>
-    <value>hdfs://nn1-clusterx.example.com:8020/home</value>
-  </property>
-  <property>
-    <name>fs.viewfs.mounttable.ClusterX.link./tmp</name>
-    <value>hdfs://nn1-clusterx.example.com:8020/tmp</value>
-  </property>
-  <property>
-    <name>fs.viewfs.mounttable.ClusterX.link./projects/foo</name>
-    <value>hdfs://nn2-clusterx.example.com:8020/projects/foo</value>
-  </property>
-  <property>
-    <name>fs.viewfs.mounttable.ClusterX.link./projects/bar</name>
-    <value>hdfs://nn3-clusterx.example.com:8020/projects/bar</value>
-  </property>
-</configuration>
-+-----------------


Mime
View raw message