hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [incubator-hudi] zhedoubushishi commented on issue #1226: [HUDI-238] Make Hudi support Scala 2.12
Date Fri, 17 Jan 2020 07:43:16 GMT
zhedoubushishi commented on issue #1226: [HUDI-238] Make Hudi support Scala 2.12
URL: https://github.com/apache/incubator-hudi/pull/1226#issuecomment-575513618
 
 
   > > Also there is another way to do this: I can rename artifact Id, e.g.`hudi-spark`
to `hudi-spark_${scala.binary.version}`, and thus avoid using `dev/change-scala-version 2.12`.
I am not sure why Spark does not choose this way.
   > 
   > @zhedoubushishi : If the artifact id renaming to hudi-spark_${scala.binary.version}
works, please go for it and make the change. I will take a look at this PR tomorrow morning
PST again.
   
   Sorry I just found ```hudi-spark_${scala.binary.version}``` might not work. 
   See the dependency tree of hudi-cli (hudi-cli depends on hudi-utilities, and hudi-utilities
depends on hudi-spark and other spark libraries):
   ```
   $ mvn dependency:tree -Pscala-2.12 -pl hudi-cli
   
   [INFO] ----------------------< org.apache.hudi:hudi-cli >----------------------
   [INFO] Building hudi-cli 0.5.1-SNAPSHOT
   [INFO] --------------------------------[ jar ]---------------------------------
   [INFO] 
   [INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ hudi-cli ---
   [INFO] org.apache.hudi:hudi-cli:jar:0.5.1-SNAPSHOT
   [INFO] +- org.scala-lang:scala-library:jar:2.12.10:compile
   [INFO] +- org.apache.hudi:hudi-hive:jar:0.5.1-SNAPSHOT:compile
   [INFO] |  +- org.apache.hudi:hudi-hadoop-mr:jar:0.5.1-SNAPSHOT:compile
   [INFO] |  \- com.beust:jcommander:jar:1.72:compile
   ...
   [INFO] +- org.apache.hudi:hudi-utilities_2.12:jar:0.5.1-SNAPSHOT:compile
   [INFO] |  +- org.apache.hudi:hudi-spark_2.11:jar:0.5.1-SNAPSHOT:compile
   [INFO] |  +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.6.7.1:compile
   [INFO] |  |  +- org.scala-lang:scala-reflect:jar:2.11.8:compile
   [INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.7.9:compile
   [INFO] |  +- org.apache.spark:spark-streaming_2.11:jar:2.4.4:compile
   [INFO] |  |  +- org.apache.spark:spark-core_2.11:jar:2.4.4:compile
   [INFO] |  |  |  +- com.twitter:chill_2.11:jar:0.9.3:compile
   [INFO] |  |  |  +- org.apache.spark:spark-launcher_2.11:jar:2.4.4:compile
   [INFO] |  |  |  +- org.apache.spark:spark-kvstore_2.11:jar:2.4.4:compile
   [INFO] |  |  |  +- org.apache.spark:spark-network-common_2.11:jar:2.4.4:compile
   [INFO] |  |  |  +- org.apache.spark:spark-network-shuffle_2.11:jar:2.4.4:compile
   [INFO] |  |  |  +- org.apache.spark:spark-unsafe_2.11:jar:2.4.4:compile
   [INFO] |  |  |  \- org.json4s:json4s-jackson_2.11:jar:3.5.3:compile
   [INFO] |  |  |     \- org.json4s:json4s-core_2.11:jar:3.5.3:compile
   [INFO] |  |  |        +- org.json4s:json4s-ast_2.11:jar:3.5.3:compile
   [INFO] |  |  |        +- org.json4s:json4s-scalap_2.11:jar:3.5.3:compile
   [INFO] |  |  |        \- org.scala-lang.modules:scala-xml_2.11:jar:1.0.6:compile
   [INFO] |  |  \- org.apache.spark:spark-tags_2.11:jar:2.4.4:compile
   [INFO] |  +- org.apache.spark:spark-streaming-kafka-0-10_2.11:jar:2.4.4:compile
   [INFO] |  |  \- org.apache.kafka:kafka-clients:jar:2.0.0:compile
   [INFO] |  +- org.apache.spark:spark-streaming-kafka-0-10_2.11:jar:tests:2.4.4:compile
   [INFO] |  +- org.antlr:stringtemplate:jar:4.0.2:compile
   [INFO] |  |  \- org.antlr:antlr-runtime:jar:3.3:compile
   [INFO] |  +- com.twitter:bijection-avro_2.11:jar:0.9.3:compile
   [INFO] |  |  +- com.twitter:bijection-core_2.11:jar:0.9.3:compile
   [INFO] |  |  \- org.scoverage:scalac-scoverage-runtime_2.11:jar:1.3.0:compile
   [INFO] |  +- io.confluent:kafka-avro-serializer:jar:3.0.0:compile
   [INFO] |  +- io.confluent:common-config:jar:3.0.0:compile
   [INFO] |  +- io.confluent:common-utils:jar:3.0.0:compile
   [INFO] |  |  \- com.101tec:zkclient:jar:0.5:compile
   [INFO] |  +- io.confluent:kafka-schema-registry-client:jar:3.0.0:compile
   [INFO] |  \- org.apache.httpcomponents:httpcore:jar:4.3.2:compile
   ```
   
   Although scala-2.12 profile overrides the ```hudi-utilities_${scala.binary.version}```
to ```hudi-utilities_2.12```, but when it comes to the dependency of hudi-utilities_2.12,
it seems maven would only use the default ```scala.binary.version``` inside ```hudi-utilities_${scala.binary.version}```
which is 2.11. And thus all the dependencies of  ```hudi-utilities_${scala.binary.version}```
goes will ```xxx_2.11```.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message