hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-13692) hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts.
Date Mon, 24 Apr 2017 19:26:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15575140#comment-15575140
] 

Steve Loughran edited comment on HADOOP-13692 at 4/24/17 7:25 PM:
------------------------------------------------------------------

just an FYI, this broke *my own* unit tests on the SPARK-7481 branch

{code}
2016-10-13 18:51:13,888 [ScalaTest-main] INFO  cloud.CloudSuite (Logging.scala:logInfo(54))
- Loading configuration from ../../cloud.xml
2016-10-13 18:51:14,214 [ScalaTest-main] WARN  util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62))
- Unable to load native-hadoop library for your platform... using builtin-java classes where
applicable
*** RUN ABORTED ***
  java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z
  at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:537)
  at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:448)
  at com.amazonaws.util.json.Jackson.<clinit>(Jackson.java:32)
  at com.amazonaws.internal.config.InternalConfig.loadfrom(InternalConfig.java:230)
  at com.amazonaws.internal.config.InternalConfig.load(InternalConfig.java:247)
  at com.amazonaws.internal.config.InternalConfig$Factory.<clinit>(InternalConfig.java:282)
  at com.amazonaws.util.VersionInfoUtils.userAgent(VersionInfoUtils.java:139)
  at com.amazonaws.util.VersionInfoUtils.initializeUserAgent(VersionInfoUtils.java:134)
  at com.amazonaws.util.VersionInfoUtils.getUserAgent(VersionInfoUtils.java:95)
  at com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:42)
  ...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
{code}

But this problem goes away in spark-assembly, the release of spark, etc. Purely this module.
Which is why I didn't catch this earlier as the system integration tests were all happy.

cause: 

# there's a newer version of jackson in use in spark (2.6.5)
# which overrides the declarations of {{jackson-annotations}} and {{jackson-databind}} under
hadoop-aws
# and which have transitive dependencies on jackson-common.
# the explicitdeclaration of jackson-common has pulled that reference one step up the dependency
graph (i.e. from under spark-cloud/hadoop-aws/amazon-aws/jackson-common.jar) to spark-cloud/hadoop-aws/jackson-common.jar.
# which gives the hadoop-aws version precedence over the one transitively referenced by the
(overridde) jackson-annotations, pulled in directly from spark-core JAR.
# so creating a version inconsistency which surfaces during test runs.

The problem isn't in spark-assembly.jar as it refers to spark-core jar directly, plcking that
version up instead.

Essentially: the fact that maven uses closest-version first in its version resolution policy
means that the depth of transitive dependencies controls whether things run or not; the explicit
declaration of the dependency was enough to cause this to surface.

Fix: explicitly exclude the hadoop-aws jackson dependencies, as was already done for hadoop-azure.


This is not me faulting my own work (how would I!), only showing that you do need to be careful
across projects as to what transitive stuff you pull in, as it turns out to be incredibly
brittle. We didn't change the jackson version here, only made that choice explicit, and a
downstream test suite fails.


was (Author: stevel@apache.org):
just an FYI, this broke *my own* unit tests on the SPARK-1481 branch

{code}
2016-10-13 18:51:13,888 [ScalaTest-main] INFO  cloud.CloudSuite (Logging.scala:logInfo(54))
- Loading configuration from ../../cloud.xml
2016-10-13 18:51:14,214 [ScalaTest-main] WARN  util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62))
- Unable to load native-hadoop library for your platform... using builtin-java classes where
applicable
*** RUN ABORTED ***
  java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z
  at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:537)
  at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:448)
  at com.amazonaws.util.json.Jackson.<clinit>(Jackson.java:32)
  at com.amazonaws.internal.config.InternalConfig.loadfrom(InternalConfig.java:230)
  at com.amazonaws.internal.config.InternalConfig.load(InternalConfig.java:247)
  at com.amazonaws.internal.config.InternalConfig$Factory.<clinit>(InternalConfig.java:282)
  at com.amazonaws.util.VersionInfoUtils.userAgent(VersionInfoUtils.java:139)
  at com.amazonaws.util.VersionInfoUtils.initializeUserAgent(VersionInfoUtils.java:134)
  at com.amazonaws.util.VersionInfoUtils.getUserAgent(VersionInfoUtils.java:95)
  at com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:42)
  ...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
{code}

But this problem goes away in spark-assembly, the release of spark, etc. Purely this module.
Which is why I didn't catch this earlier as the system integration tests were all happy.

cause: 

# there's a newer version of jackson in use in spark (2.6.5)
# which overrides the declarations of {{jackson-annotations}} and {{jackson-databind}} under
hadoop-aws
# and which have transitive dependencies on jackson-common.
# the explicitdeclaration of jackson-common has pulled that reference one step up the dependency
graph (i.e. from under spark-cloud/hadoop-aws/amazon-aws/jackson-common.jar) to spark-cloud/hadoop-aws/jackson-common.jar.
# which gives the hadoop-aws version precedence over the one transitively referenced by the
(overridde) jackson-annotations, pulled in directly from spark-core JAR.
# so creating a version inconsistency which surfaces during test runs.

The problem isn't in spark-assembly.jar as it refers to spark-core jar directly, plcking that
version up instead.

Essentially: the fact that maven uses closest-version first in its version resolution policy
means that the depth of transitive dependencies controls whether things run or not; the explicit
declaration of the dependency was enough to cause this to surface.

Fix: explicitly exclude the hadoop-aws jackson dependencies, as was already done for hadoop-azure.


This is not me faulting my own work (how would I!), only showing that you do need to be careful
across projects as to what transitive stuff you pull in, as it turns out to be incredibly
brittle. We didn't change the jackson version here, only made that choice explicit, and a
downstream test suite fails.

> hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath
conflicts.
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-13692
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13692
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>            Priority: Minor
>             Fix For: 2.8.0, 3.0.0-alpha2
>
>         Attachments: HADOOP-13692-branch-2.001.patch
>
>
> If an end user's application has a dependency on hadoop-aws and no other Hadoop artifacts,
then it picks up a transitive dependency on Jackson 2.5.3 jars through the AWS SDK.  This
can cause conflicts at deployment time, because Hadoop has a dependency on version 2.2.3,
and the 2 versions are not compatible with one another.  We can prevent this problem by changing
hadoop-aws to declare explicit dependencies on the Jackson artifacts, at the version Hadoop
wants.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message