crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-244) Add support for hadoop1 and hadoop2 classifiers to the POM and release artifacts
Date Thu, 25 Jul 2013 15:43:50 GMT


Tom White commented on CRUNCH-244:

In Avro we create three artifacts, one with the hadoop1 classifier, one with hadoop2, and
one without a classifier (identical to the hadoop1 JAR). E.g.

> The second time it would fail to deploy because the crunch-core-0.7 already existed.

The way we deploy these is to run the hadoop2 profile first which actually creates the hadoop2
artifact, and one without a classifier (but containing hadoop2 classes), then we run the hadoop1
profile so that the one without a classifier is overwritten with hadoop1 classes. Deploying
to the Nexus repo twice like this worked OK. See instructions under the "Building" section

> I also think that if you use the secondary artifacts to group them all into a single
groupId/artifactId/version, the POM file would only be describing one set of the dependencies
and not the appropriate one for the version of hadoop you are compiling against.

That's true. In Avro it's OK since it's only the avro-mapred JAR we need to treat like this,
but in Crunch it is the core JAR which other artifacts in the same project depend on. In Parquet
we used reflection to avoid having to have two binary artifacts, but unfortunately this would
not be acceptable in Crunch for [performance reasons|].

> Would it be better if we actually changed the version in each profile?

I think so.
> Add support for hadoop1 and hadoop2 classifiers to the POM and release artifacts
> --------------------------------------------------------------------------------
>                 Key: CRUNCH-244
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>            Reporter: Josh Wills
>         Attachments: CRUNCH-244.patch
> We would like to be able to publish jars and binary release artifacts that are compiled
against both the hadoop1 as well as the hadoop2 APIs as part of doing a release in order to
make the transition easier for clients to use.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message