spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dongjoon-hyun <...@git.apache.org>
Subject [GitHub] spark pull request #20704: [SPARK-23551][BUILD] Exclude `hadoop-mapreduce-cl...
Date Thu, 01 Mar 2018 18:01:39 GMT
GitHub user dongjoon-hyun opened a pull request:

    https://github.com/apache/spark/pull/20704

    [SPARK-23551][BUILD] Exclude `hadoop-mapreduce-client-core` dependency from `orc-mapreduce`

    ## What changes were proposed in this pull request?
    
    This issue aims to prevent `orc-mapreduce` dependency makes IDEs and maven confused.
    
    **BEFORE**
    Please note that `2.6.4`.
    ```
    $ mvn dependency:tree -Phadoop-2.7 -Dincludes=org.apache.hadoop:hadoop-mapreduce-client-core
    ...
    [INFO] ------------------------------------------------------------------------
    [INFO] Building Spark Project Catalyst 2.4.0-SNAPSHOT
    [INFO] ------------------------------------------------------------------------
    [INFO]
    [INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ spark-catalyst_2.11 ---
    [INFO] org.apache.spark:spark-catalyst_2.11:jar:2.4.0-SNAPSHOT
    [INFO] \- org.apache.spark:spark-core_2.11:jar:2.4.0-SNAPSHOT:compile
    [INFO]    \- org.apache.hadoop:hadoop-client:jar:2.7.3:compile
    [INFO]       \- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3:compile
    [INFO]
    [INFO] ------------------------------------------------------------------------
    [INFO] Building Spark Project SQL 2.4.0-SNAPSHOT
    [INFO] ------------------------------------------------------------------------
    [INFO]
    [INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ spark-sql_2.11 ---
    [INFO] org.apache.spark:spark-sql_2.11:jar:2.4.0-SNAPSHOT
    [INFO] \- org.apache.orc:orc-mapreduce:jar:nohive:1.4.3:compile
    [INFO]    \- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.4:compile
    ```
    
    **AFTER**
    ```
    $ mvn dependency:tree -Phadoop-2.7 -Dincludes=org.apache.hadoop:hadoop-mapreduce-client-core
    ...
    [INFO] ------------------------------------------------------------------------
    [INFO] Building Spark Project Catalyst 2.4.0-SNAPSHOT
    [INFO] ------------------------------------------------------------------------
    [INFO]
    [INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ spark-catalyst_2.11 ---
    [INFO] org.apache.spark:spark-catalyst_2.11:jar:2.4.0-SNAPSHOT
    [INFO] \- org.apache.spark:spark-core_2.11:jar:2.4.0-SNAPSHOT:compile
    [INFO]    \- org.apache.hadoop:hadoop-client:jar:2.7.3:compile
    [INFO]       \- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3:compile
    [INFO]
    [INFO] ------------------------------------------------------------------------
    [INFO] Building Spark Project SQL 2.4.0-SNAPSHOT
    [INFO] ------------------------------------------------------------------------
    [INFO]
    [INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ spark-sql_2.11 ---
    [INFO] org.apache.spark:spark-sql_2.11:jar:2.4.0-SNAPSHOT
    [INFO] \- org.apache.spark:spark-core_2.11:jar:2.4.0-SNAPSHOT:compile
    [INFO]    \- org.apache.hadoop:hadoop-client:jar:2.7.3:compile
    [INFO]       \- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3:compile
    ```
    
    ## How was this patch tested?
    
    1. Pass the Jenkins with `dev/test-dependencies.sh` with the existing dependencies.
    2. Manually do the following and see the change.
    ```
    mvn dependency:tree -Phadoop-2.7 -Dincludes=org.apache.hadoop:hadoop-mapreduce-client-core
    ```

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dongjoon-hyun/spark SPARK-23551

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20704.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20704
    
----
commit dbb5ae504786ede4a336faa4033809a63ec10f92
Author: Dongjoon Hyun <dongjoon@...>
Date:   2018-03-01T17:55:50Z

    [SPARK-23551][BUILD] Exclude `hadoop-mapreduce-client-core` dependency from `orc-mapreduce`

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message