hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yjzhan...@apache.org
Subject hadoop git commit: HDFS-9638. Improve DistCp Help and documentation. (Wei-Chiu Chuang via Yongjun Zhang)
Date Fri, 29 Jan 2016 20:22:21 GMT
Repository: hadoop
Updated Branches:
  refs/heads/branch-2.8 eb2366eda -> f028e3d7b


HDFS-9638. Improve DistCp Help and documentation. (Wei-Chiu Chuang via Yongjun Zhang)

(cherry picked from commit eddd823cd6246ddc66218eb01009c44b0236eaaa)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f028e3d7
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f028e3d7
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f028e3d7

Branch: refs/heads/branch-2.8
Commit: f028e3d7bcce9dd8428beeffda2cb878b9ef04ce
Parents: eb2366e
Author: Yongjun Zhang <yzhang@cloudera.com>
Authored: Fri Jan 29 12:11:55 2016 -0800
Committer: Yongjun Zhang <yzhang@cloudera.com>
Committed: Fri Jan 29 12:13:18 2016 -0800

----------------------------------------------------------------------
 hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt                  | 3 +++
 .../java/org/apache/hadoop/tools/DistCpOptionSwitch.java     | 2 +-
 hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm    | 8 +++++---
 .../test/java/org/apache/hadoop/tools/TestOptionsParser.java | 3 ++-
 4 files changed, 11 insertions(+), 5 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/f028e3d7/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index 4385aed..d0813c4 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -891,6 +891,9 @@ Release 2.8.0 - UNRELEASED
     HDFS-9706. Log more details in debug logs in BlockReceiver's constructor.
     (Xiao Chen via Yongjun Zhang)
 
+    HDFS-9638. Improve DistCp Help and documentation.
+    (Wei-Chiu Chuang via Yongjun Zhang)
+
   OPTIMIZATIONS
 
     HDFS-8026. Trace FSOutputSummer#writeChecksumChunks rather than

http://git-wip-us.apache.org/repos/asf/hadoop/blob/f028e3d7/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
index ed4a0b2..ef6b2c1 100644
--- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
+++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
@@ -82,7 +82,7 @@ public enum DistCpOptionSwitch {
    */
   SSL_CONF(DistCpConstants.CONF_LABEL_SSL_CONF,
       new Option("mapredSslConf", true, "Configuration for ssl config file" +
-          ", to use with hftps://")),
+          ", to use with hftps://. Must be in the classpath.")),
   /**
    * Number of threads for building source file listing (before map-reduce
    * phase, max one listStatus per thread at a time).

http://git-wip-us.apache.org/repos/asf/hadoop/blob/f028e3d7/hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm b/hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm
index 7fa1747..31f0444 100644
--- a/hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm
+++ b/hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm
@@ -218,7 +218,7 @@ Command Line Options
 
 Flag              | Description                          | Notes
 ----------------- | ------------------------------------ | --------
-`-p[rbugpcax]` | Preserve r: replication number b: block size u: user g: group p: permission
c: checksum-type a: ACL x: XAttr | Modification times are not preserved. Also, when `-update`
is specified, status updates will **not** be synchronized unless the file sizes also differ
(i.e. unless the file is re-created). If -pa is specified, DistCp preserves the permissions
also because ACLs are a super-set of permissions.
+`-p[rbugpcaxt]` | Preserve r: replication number b: block size u: user g: group p: permission
c: checksum-type a: ACL x: XAttr t: timestamp | When `-update` is specified, status updates
will **not** be synchronized unless the file sizes also differ (i.e. unless the file is re-created).
If -pa is specified, DistCp preserves the permissions also because ACLs are a super-set of
permissions.
 `-i` | Ignore failures | As explained in the Appendix, this option will keep more accurate
statistics about the copy than the default case. It also preserves logs from failed copies,
which can be valuable for debugging. Finally, a failing map will not cause the job to fail
before all splits are attempted.
 `-log <logdir>` | Write logs to \<logdir\> | DistCp keeps logs of each file it
attempts to copy as map output. If a map fails, the log output will not be retained if it
is re-executed.
 `-m <num_maps>` | Maximum number of simultaneous copies | Specify the number of maps
to copy data. Note that more maps may not necessarily improve throughput.
@@ -234,6 +234,9 @@ Flag              | Description                          | Notes
 `-atomic {-tmp <tmp_dir>}` | Specify atomic commit, with optional tmp directory. |
`-atomic` instructs DistCp to copy the source data to a temporary target location, and then
move the temporary target to the final-location atomically. Data will either be available
at final target in a complete and consistent form, or not at all. Optionally, `-tmp` may be
used to specify the location of the tmp-target. If not specified, a default is chosen. **Note:**
tmp_dir must be on the final target cluster.
 `-mapredSslConf <ssl_conf_file>` | Specify SSL Config file, to be used with HSFTP source
| When using the hsftp protocol with a source, the security- related properties may be specified
in a config-file and passed to DistCp. \<ssl_conf_file\> needs to be in the classpath.
 `-async` | Run DistCp asynchronously. Quits as soon as the Hadoop Job is launched. | The
Hadoop Job-id is logged, for tracking.
+`-diff` | Use snapshot diff report to identify the difference between source and target.
|
+`-numListstatusThreads` | Number of threads to use for building file listing | At most 40
threads.
+`-skipcrccheck` | Whether to skip CRC checks between source and target paths. |
 
 Architecture of DistCp
 ----------------------
@@ -441,8 +444,7 @@ $H3 SSL Configurations for HSFTP sources
   * `ssl.client.truststore.password`: (Optional) Password for the trust-store
     file.
 
-  The following is an example of the contents of the contents of a SSL
-  Configuration file:
+  The following is an example SSL configuration file:
 
     <configuration>
       <property>

http://git-wip-us.apache.org/repos/asf/hadoop/blob/f028e3d7/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
b/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
index 4dc08fb..8d03319 100644
--- a/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
+++ b/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
@@ -501,7 +501,7 @@ public class TestOptionsParser {
     Assert.assertFalse(options.shouldPreserve(FileAttribute.XATTR));
 
     options = OptionsParser.parse(new String[] {
-        "-pbrgupcax",
+        "-pbrgupcaxt",
         "-f",
         "hdfs://localhost:8020/source/first",
         "hdfs://localhost:8020/target/"});
@@ -513,6 +513,7 @@ public class TestOptionsParser {
     Assert.assertTrue(options.shouldPreserve(FileAttribute.CHECKSUMTYPE));
     Assert.assertTrue(options.shouldPreserve(FileAttribute.ACL));
     Assert.assertTrue(options.shouldPreserve(FileAttribute.XATTR));
+    Assert.assertTrue(options.shouldPreserve(FileAttribute.TIMES));
 
     options = OptionsParser.parse(new String[] {
         "-pc",


Mime
View raw message