incubator-crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabriel Reid (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-16) Fix dependency versions
Date Tue, 21 Aug 2012 19:37:38 GMT


Gabriel Reid commented on CRUNCH-16:

I've realized that this is directly related to something else I've been working on with Crunch
where I'm combining reflect and specific data, so I took a second look at doing automatic

As far as I know/recall, the relevant issue that was fixed in Avro 1.7.0 was AVRO-1046, which
was about the ReflectDatumReader properly handling specific data; this was important because
the ReflectDatumReader is used as the fallback anytime there is reflect data included.

The attached patch fixes a small bug in the existing patches (the ReflectDatumReader wouldn't
be used if the CAN_COMBINE_SPECIFIC_AND_REFLECT_SCHEMAS bit was set), and it also includes
code to dynamically initialize the CAN_COMBINE_SPECIFIC_AND_REFLECT_SCHEMAS (which is now
final). Can you take a look and let me know what you think? I've run the integration tests
on Avro 1.5.4, 1.6.2, and 1.7.0 and it works on all of them.
> Fix dependency versions
> -----------------------
>                 Key: CRUNCH-16
>                 URL:
>             Project: Crunch
>          Issue Type: Task
>            Reporter: Matthias Friedrich
>            Assignee: Matthias Friedrich
>            Priority: Critical
>              Labels: maven
>         Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch, CRUNCH-16-avro-preparation.patch,
CRUNCH-16-avro-reflect-specific-dynamic-init.patch.txt, CRUNCH-16-avro-reflect-specific-static-init.patch
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For
example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3.
This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that
we don't ship artifacts that are already present on the runtime classpath. If our dependencies
introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message