incubator-crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabriel Reid (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-16) Fix dependency versions
Date Tue, 21 Aug 2012 19:37:38 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438976#comment-13438976
] 

Gabriel Reid commented on CRUNCH-16:
------------------------------------

I've realized that this is directly related to something else I've been working on with Crunch
where I'm combining reflect and specific data, so I took a second look at doing automatic
initialization of the CAN_COMBINE_SPECIFIC_AND_REFLECT_SCHEMAS.

As far as I know/recall, the relevant issue that was fixed in Avro 1.7.0 was AVRO-1046, which
was about the ReflectDatumReader properly handling specific data; this was important because
the ReflectDatumReader is used as the fallback anytime there is reflect data included.

The attached patch fixes a small bug in the existing patches (the ReflectDatumReader wouldn't
be used if the CAN_COMBINE_SPECIFIC_AND_REFLECT_SCHEMAS bit was set), and it also includes
code to dynamically initialize the CAN_COMBINE_SPECIFIC_AND_REFLECT_SCHEMAS (which is now
final). Can you take a look and let me know what you think? I've run the integration tests
on Avro 1.5.4, 1.6.2, and 1.7.0 and it works on all of them.
                
> Fix dependency versions
> -----------------------
>
>                 Key: CRUNCH-16
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-16
>             Project: Crunch
>          Issue Type: Task
>            Reporter: Matthias Friedrich
>            Assignee: Matthias Friedrich
>            Priority: Critical
>              Labels: maven
>         Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch, CRUNCH-16-avro-preparation.patch,
CRUNCH-16-avro-reflect-specific-dynamic-init.patch.txt, CRUNCH-16-avro-reflect-specific-static-init.patch
>
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For
example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3.
This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that
we don't ship artifacts that are already present on the runtime classpath. If our dependencies
introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message