spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sro...@apache.org
Subject spark git commit: [SPARK-25330][BUILD][BRANCH-2.3] Revert Hadoop 2.7 to 2.7.3
Date Fri, 07 Sep 2018 04:41:17 GMT
Repository: spark
Updated Branches:
  refs/heads/master 1b1711e05 -> b0ada7dce


[SPARK-25330][BUILD][BRANCH-2.3] Revert Hadoop 2.7 to 2.7.3

## What changes were proposed in this pull request?
How to reproduce permission issue:
```sh
# build spark
./dev/make-distribution.sh --name SPARK-25330 --tgz  -Phadoop-2.7 -Phive -Phive-thriftserver
-Pyarn

tar -zxf spark-2.4.0-SNAPSHOT-bin-SPARK-25330.tar && cd spark-2.4.0-SNAPSHOT-bin-SPARK-25330
export HADOOP_PROXY_USER=user_a
bin/spark-sql

export HADOOP_PROXY_USER=user_b
bin/spark-sql
```
```java
Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException:
Permission denied: user=user_b, access=EXECUTE, inode="/tmp/hive-$%7Buser.name%7D/user_b/668748f2-f6c5-4325-a797-fd0a7ee7f4d4":user_b:hadoop:drwx------
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
```

The issue occurred in this commit: https://github.com/apache/hadoop/commit/feb886f2093ea5da0cd09c69bd1360a335335c86.
This pr revert Hadoop 2.7 to 2.7.3 to avoid this issue.

## How was this patch tested?
unit tests and manual tests.

Closes #22327 from wangyum/SPARK-25330.

Authored-by: Yuming Wang <yumwang@ebay.com>
Signed-off-by: Sean Owen <sean.owen@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b0ada7dc
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b0ada7dc
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b0ada7dc

Branch: refs/heads/master
Commit: b0ada7dce02d101b6a04323d8185394e997caca4
Parents: 1b1711e
Author: Yuming Wang <yumwang@ebay.com>
Authored: Thu Sep 6 21:41:13 2018 -0700
Committer: Sean Owen <sean.owen@databricks.com>
Committed: Thu Sep 6 21:41:13 2018 -0700

----------------------------------------------------------------------
 assembly/README                |  2 +-
 dev/deps/spark-deps-hadoop-2.7 | 31 +++++++++++++++----------------
 docs/building-spark.md         |  2 +-
 pom.xml                        |  2 +-
 4 files changed, 18 insertions(+), 19 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/b0ada7dc/assembly/README
----------------------------------------------------------------------
diff --git a/assembly/README b/assembly/README
index affd281..d5dafab 100644
--- a/assembly/README
+++ b/assembly/README
@@ -9,4 +9,4 @@ This module is off by default. To activate it specify the profile in the command
 
 If you need to build an assembly for a different version of Hadoop the
 hadoop-version system property needs to be set as in this example:
-  -Dhadoop.version=2.7.7
+  -Dhadoop.version=2.7.3

http://git-wip-us.apache.org/repos/asf/spark/blob/b0ada7dc/dev/deps/spark-deps-hadoop-2.7
----------------------------------------------------------------------
diff --git a/dev/deps/spark-deps-hadoop-2.7 b/dev/deps/spark-deps-hadoop-2.7
index 5e12ca0..dcb5d63 100644
--- a/dev/deps/spark-deps-hadoop-2.7
+++ b/dev/deps/spark-deps-hadoop-2.7
@@ -64,21 +64,21 @@ gson-2.2.4.jar
 guava-14.0.1.jar
 guice-3.0.jar
 guice-servlet-3.0.jar
-hadoop-annotations-2.7.7.jar
-hadoop-auth-2.7.7.jar
-hadoop-client-2.7.7.jar
-hadoop-common-2.7.7.jar
-hadoop-hdfs-2.7.7.jar
-hadoop-mapreduce-client-app-2.7.7.jar
-hadoop-mapreduce-client-common-2.7.7.jar
-hadoop-mapreduce-client-core-2.7.7.jar
-hadoop-mapreduce-client-jobclient-2.7.7.jar
-hadoop-mapreduce-client-shuffle-2.7.7.jar
-hadoop-yarn-api-2.7.7.jar
-hadoop-yarn-client-2.7.7.jar
-hadoop-yarn-common-2.7.7.jar
-hadoop-yarn-server-common-2.7.7.jar
-hadoop-yarn-server-web-proxy-2.7.7.jar
+hadoop-annotations-2.7.3.jar
+hadoop-auth-2.7.3.jar
+hadoop-client-2.7.3.jar
+hadoop-common-2.7.3.jar
+hadoop-hdfs-2.7.3.jar
+hadoop-mapreduce-client-app-2.7.3.jar
+hadoop-mapreduce-client-common-2.7.3.jar
+hadoop-mapreduce-client-core-2.7.3.jar
+hadoop-mapreduce-client-jobclient-2.7.3.jar
+hadoop-mapreduce-client-shuffle-2.7.3.jar
+hadoop-yarn-api-2.7.3.jar
+hadoop-yarn-client-2.7.3.jar
+hadoop-yarn-common-2.7.3.jar
+hadoop-yarn-server-common-2.7.3.jar
+hadoop-yarn-server-web-proxy-2.7.3.jar
 hk2-api-2.4.0-b34.jar
 hk2-locator-2.4.0-b34.jar
 hk2-utils-2.4.0-b34.jar
@@ -117,7 +117,6 @@ jersey-guava-2.22.2.jar
 jersey-media-jaxb-2.22.2.jar
 jersey-server-2.22.2.jar
 jetty-6.1.26.jar
-jetty-sslengine-6.1.26.jar
 jetty-util-6.1.26.jar
 jline-2.14.6.jar
 joda-time-2.9.3.jar

http://git-wip-us.apache.org/repos/asf/spark/blob/b0ada7dc/docs/building-spark.md
----------------------------------------------------------------------
diff --git a/docs/building-spark.md b/docs/building-spark.md
index 1d3e0b1..1501f0b 100644
--- a/docs/building-spark.md
+++ b/docs/building-spark.md
@@ -67,7 +67,7 @@ Examples:
     ./build/mvn -Pyarn -DskipTests clean package
 
     # Apache Hadoop 2.7.X and later
-    ./build/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.7 -DskipTests clean package
+    ./build/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.3 -DskipTests clean package
 
 ## Building With Hive and JDBC Support
 

http://git-wip-us.apache.org/repos/asf/spark/blob/b0ada7dc/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index da526a1..05e3b05 100644
--- a/pom.xml
+++ b/pom.xml
@@ -2683,7 +2683,7 @@
     <profile>
       <id>hadoop-2.7</id>
       <properties>
-        <hadoop.version>2.7.7</hadoop.version>
+        <hadoop.version>2.7.3</hadoop.version>
         <curator.version>2.7.1</curator.version>
       </properties>
     </profile>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message