camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (CAMEL-7249) Upgrade Camel-HDFS component to be compatible with Apache Hadoop 2.x
Date Wed, 12 Mar 2014 09:12:45 GMT


ASF GitHub Bot commented on CAMEL-7249:

GitHub user grgrzybek opened a pull request:

    [CAMEL-7249] camel-hdfs2 component

    Finally I've managed to adapt camel-hdfs component to work with hadoop-2.3.0 libraries.
    Thanks to Jean-Baptiste Onofré for providing hadoop-client-2.3.0 OSGi bundle. It'd be
hard to use OSGi bundle per hadoop-*.jar artifact because of split packages. So aggregate
`hadoop-client` is ok.
    * Generally camel-hdfs2 component is repackaged version of camel-hdfs component with Maven
dependencies switched to hadoop-common:2.3.0 and hadoop-hdfs:2.3.0
    * Some deprecated API usage was replaced by 2.3.0 equivalents
    * This component works in OSGi environment (e.g., inside Apache Karaf) both as classic
bundle and blueprint definition
    * There are some quirks related to OSGi:
    * - Hadoop uses `java.util.ServiceLocator` to find filesystem implementations, so some
static code initialization must be performed to replace this mechanism (works both in pure
OSGi and Blueprint container)
    * - `Import-Package` is not just a list generated by *maven-bundle-plugin* - it must be
adjusted to contain all packages reachable by reflection (e.g. `org.apache.hadoop.hdfs*`)
    * - I didn't touch some bundle versions in parent POM and instead I used hardcoded versions
for example in AVRO in `platforms/karaf/features/src/main/resources/features.xml`, as the
one used in parent POM requires OSGi package `sun.misc`
    * - I had to tweak `org.osgi.framework.system.packages.extra` property in `camel-itest-osgi/**/itest/karaf/`
to contain Xerces packages
    * The documentation must be added for new component

You can merge this pull request into a Git repository by running:

    $ git pull CAMEL-7249

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #110
commit af7661abb98b85658c35974bb914aae6602365a4
Author: Grzegorz Grzybek <>
Date:   2014-03-10T14:05:43Z

    [CAMEL-7249] Working version of camel-hdfs2 component
    * All unit tests works fine.
    * Usage of all deprecated APIs changed.
    * hdfs://localhost tests work (@Ignored - they require external hdfs

commit 0b9b3531e4a4bfe82492e3037eb36381ad0b1865
Author: Grzegorz Grzybek <>
Date:   2014-03-11T18:29:30Z

    [CAMEL-7249] Working OSGi tests for camel-hdfs2
    There are some issues with this implementation:
    * Import-Packages for `camel-hdfs2` are not just the ones generated by
      maven-bundle plugin. They must contain some packages reachable only by
      reflection (different FileSystem implementations)
    * There is problem with using java.util.ServiceLoader inside OSGi and Hadoop 2
      uses ServiceLoader to discover different FileSystem implementations. For OSGi
      (especially blueprint container) we have to use a hack to populate STATIC
    * `camel-hdfs2` feature contains some hardcoded versions and for e.g., AVRO a
      downgraded one (1.7.5_1 requires `sun.misc` package - this changed however
      since SMX4-1709)
    * org.osgi.framework.system.packages.extra property in
      camel-itest-osgi/**/itest/karaf/ was added with xerces
    * documentation has to be added describing hadoop2 specific configuration inside
      OSGi - this changed since hadoop 1.2.1 (using java.util.ServiceLoader)

commit 2131dce4d8df10ec207d74f160e926dd74f068c5
Author: Grzegorz Grzybek <>
Date:   2014-03-12T07:05:36Z

    [CAMEL-7249] Working camel-itest-karaf for camel-hdfs2

commit cd101646c688206f73adac68352d14d264287b30
Author: Grzegorz Grzybek <>
Date:   2014-03-12T08:54:29Z

    [CAMEL-7249] Working blueprint tests
    All itests now work both with LOCAL and HDFS filesystems


> Upgrade Camel-HDFS component to be compatible with Apache Hadoop 2.x
> --------------------------------------------------------------------
>                 Key: CAMEL-7249
>                 URL:
>             Project: Camel
>          Issue Type: New Feature
>            Reporter: Grzegorz Grzybek
>            Assignee: Grzegorz Grzybek
> Currently camel-hdfs depends on hadoop-core:1.2.1. Make (working name) {{camel-hdfs2}}
component using newest (hadoop 2.3.0) libraries.

This message was sent by Atlassian JIRA

View raw message