hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From xg...@apache.org
Subject hadoop git commit: MAPREDUCE-6173. Document the configuration of deploying MR over distributed cache with enabling wired encryption at the same time. Contributed by Junping Du.
Date Tue, 13 Jan 2015 18:09:14 GMT
Repository: hadoop
Updated Branches:
  refs/heads/branch-2 7a75f4eed -> 0562f382f


MAPREDUCE-6173. Document the configuration of deploying MR over
distributed cache with enabling wired encryption at the same time.
Contributed by Junping Du.

(cherry picked from commit 10ac5abb867ead7131f3c76b27e48f7908b1d7f1)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/0562f382
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/0562f382
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/0562f382

Branch: refs/heads/branch-2
Commit: 0562f382fa642980634a95447565cf6f0a317adc
Parents: 7a75f4e
Author: Xuan <xgong@apache.org>
Authored: Tue Jan 13 10:04:31 2015 -0800
Committer: Xuan <xgong@apache.org>
Committed: Tue Jan 13 10:08:24 2015 -0800

----------------------------------------------------------------------
 hadoop-mapreduce-project/CHANGES.txt            |  4 +++
 .../src/site/apt/DistributedCacheDeploy.apt.vm  | 33 ++++++++++++++++++++
 2 files changed, 37 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/0562f382/hadoop-mapreduce-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/CHANGES.txt b/hadoop-mapreduce-project/CHANGES.txt
index ea0d118..b119b46 100644
--- a/hadoop-mapreduce-project/CHANGES.txt
+++ b/hadoop-mapreduce-project/CHANGES.txt
@@ -26,6 +26,10 @@ Release 2.7.0 - UNRELEASED
     HADOOP-11032. Replace use of Guava's Stopwatch with Hadoop's StopWatch
     (ozawa)
 
+    MAPREDUCE-6173. Document the configuration of deploying MR over distributed
+    cache with enabling wired encryption at the same time. 
+    (Junping Du via xgong)
+
   OPTIMIZATIONS
 
     MAPREDUCE-6169. MergeQueue should release reference to the current item 

http://git-wip-us.apache.org/repos/asf/hadoop/blob/0562f382/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/DistributedCacheDeploy.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/DistributedCacheDeploy.apt.vm
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/DistributedCacheDeploy.apt.vm
index 9cb7403..2195e10 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/DistributedCacheDeploy.apt.vm
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/DistributedCacheDeploy.apt.vm
@@ -116,3 +116,36 @@ Hadoop MapReduce Next Generation - Distributed Cache Deploy
   change to something like the following:
 
     <<<$HADOOP_CONF_DIR,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/mapreduce/*,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/mapreduce/lib/*,$HADOOP_COMMON_HOME/share/hadoop/common/*,$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$HADOOP_YARN_HOME/share/hadoop/yarn/*,$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*>>>
+
+** NOTE: 
+
+  If shuffle encryption is also enabled in the cluster, then we could meet the problem that
MR job get failed with exception like below: 
+  
++---+
+2014-10-10 02:17:16,600 WARN [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher:
Failed to connect to junpingdu-centos5-3.cs1cloud.internal:13562 with 1 map outputs
+javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path
building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find
valid certification path to requested target
+    at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
+    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1731)
+    at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:241)
+    at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:235)
+    at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1206)
+    at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:136)
+    at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:593)
+    at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:529)
+    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:925)
+    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1170)
+    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1197)
+    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1181)
+    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:434)
+    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.setNewClient(AbstractDelegateHttpsURLConnection.java:81)
+    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.setNewClient(AbstractDelegateHttpsURLConnection.java:61)
+    at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:584)
+    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1193)
+    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
+    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:318)
+    at org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:427)
+....
+
++---+
+
+  This is because MR client (deployed from HDFS) cannot access ssl-client.xml in local FS
under directory of $HADOOP_CONF_DIR. To fix the problem, we can add the directory with ssl-client.xml
to the classpath of MR which is specified in "mapreduce.application.classpath" as mentioned
above. To avoid MR application being affected by other local configurations, it is better
to create a dedicated directory for putting ssl-client.xml, e.g. a sub-directory under $HADOOP_CONF_DIR,
like: $HADOOP_CONF_DIR/security.


Mime
View raw message