orc-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Evans <jeffrey.wayne.ev...@gmail.com>
Subject Re: Orc core (Java) dependency on hadoop-common
Date Wed, 30 Jan 2019 21:17:08 GMT
To close the loop on this admittedly old thread, we now have some code
that performs this conversion as part of our open source product.  I'm
mentioning it here in case anyone else finds it useful, or has any
feedback on implementation, bugs, invalid assumptions, etc.

This class converts an Avro schema to an ORC schema:
https://github.com/streamsets/datacollector/blob/master/mapreduce-protolib/src/main/java/com/streamsets/pipeline/lib/util/avroorc/AvroToOrcSchemaConverter.java

This class converts an Avro file to an ORC file (using a schema built
using the above):
https://github.com/streamsets/datacollector/blob/master/mapreduce-protolib/src/main/java/com/streamsets/pipeline/lib/util/avroorc/AvroToOrcRecordConverter.java

Both make use of a utility class:
https://github.com/streamsets/datacollector/blob/master/commonlib/src/main/java/com/streamsets/pipeline/lib/util/AvroTypeUtil.java

There are some test cases here:
https://github.com/streamsets/datacollector/tree/master/mapreduce-protolib/src/test/java/com/streamsets/pipeline/lib/util/avroorc

Thanks for all the info before, and any feedback/critiques are welcomed!


On Wed, Jan 17, 2018 at 4:55 PM Jeff Evans
<jeffrey.wayne.evans@gmail.com> wrote:
>
> Thanks, István and Owen!
>
> I appreciate the input.  At the moment, I'm developing against
> orc-core 1.4.1.  I think I will go the route of excluding
> hadoop-common from the orc-core dependency and explicitly scope it as
> provided.  For modules that will ultimately live in one of our Hadoop
> deployments, this should work fine.  Moreover, we already adopt this
> sort of packaging strategy in our project, so it wouldn't be too much
> of a stretch.  For the "standalone" operation, I will probably just
> create a separate module that explicitly declares hadoop-common as a
> compile dependency, so those not on Hadoop can simply bring in the
> same version that orc itself specifies.
>
> I think the longer term approach you describe makes good sense,
> István.  Unfortunately, given other priorities I wouldn't be able to
> devote any time to it in the near future.  As far as Hadoop
> versioning, I think the minimum/desired approach outlined in Owen's
> message would work fine.
>
> On Wed, Jan 17, 2018 at 3:58 PM, István <leccine@gmail.com> wrote:
> > Hi Jeff,
> >
> > Few months back I wondering about the same topic. Unfortunately dependency
> > management and importing libraries is not the strongest suit of Hadoop
> > related libraries and that includes ORC. We got with our project to the
> > point when we considered forking ORC and just create our own version of it
> > becuase we want to use it outside Hadoop. Unfortunately Hadoop related code
> > is all over the place so we decided to just exclude a bunch of libraries and
> > we ended up with a pom.xml like this:
> >
> > https://gist.github.com/l1x/0c00fe69bdcb6db305e0bffae042817c
> >
> > Keep in mind this is an older version of ORC that is included in the Hive
> > 1.2.1 release. I also started to work on a project to deal with Hadoop
> > dependencies easier but we dropped the entire project altogether.
> >
> > I think what would be reasonable is to have libraries like ORC at the bottom
> > of the dependency stack (orc-core) and create a library that provides an
> > interface for Hadoop or any project that wants to use this file format
> > (orc-hadoop, orc-something, etc.) so that we don't have this dependency hell
> > that you can see in projects like ORC. I am not sure who else is interested
> > in such a project but if you are I think I could provide you some
> > development time.
> >
> > Owen was really helpful with the efforts. See more here:
> > https://issues.apache.org/jira/browse/ORC-151
> > https://github.com/apache/orc/pull/96
> >
> > Thanks,
> > Istvan
> >
> > On Wed, Jan 17, 2018 at 6:16 PM, Jeff Evans <jeffrey.wayne.evans@gmail.com>
> > wrote:
> >>
> >> Hi,
> >>
> >> I am a software engineer with StreamSets, and am working on a project
> >> to incorporate ORC support into our product.  The first phase of this
> >> will be to support Avro to ORC conversion. (I saw a post on this topic
> >> to this list a couple months ago, before I joined.  Would be happy to
> >> share more details/code for scrutiny once it's closer to completion.)
> >>
> >> One issue I'm running into is the dependency of orc-core on
> >> hadoop-common.  Our product can be deployed in a variety of Hadoop
> >> distributions from different vendors, and also standalone (i.e. not in
> >> Hadoop at all).  Therefore, this dependency makes it difficult for us
> >> to incorporate orc-core in a central way in our codebase (since the
> >> vendor typically provides this jar in their installation).  Besides
> >> that, hadoop-common also brings in a number of other problematic
> >> dependencies for us (the deprecated com.sun.jersey group for Jersey
> >> and zookeeper, to name a couple).
> >>
> >> Does anyone have suggestions for how to work around this?  It seems
> >> the only actual classes I reference are the same ones referenced in
> >> the core-java tutorial (org.apache.hadoop.conf.Configuration and
> >> org.apache.hadoop.fs.Path), although obviously the library may be
> >> making use of more itself.  Are there any plans to remove the
> >> dependency on Hadoop down the line, or should I accommodate this by
> >> shuffling our dependencies such that our code only lives in a
> >> Hadoop-provided packaging configuration?  Any insight is appreciated.
> >
> >
> >
> >
> > --
> > the sun shines for all
> >
> >

Mime
View raw message